Object trace device, object method, and object trace program

ABSTRACT

Object-zone extraction section  12  receives image information through image input terminal  11  and extracts object zones from the received image information. Characteristic-quantity generation section  14  generates the characteristic quantities of objects and the characteristic quantities of object zones. Characteristic-quantity synthesis section  15  synthesizes the characteristic quantities of a plurality of objects to generate a synthesized characteristic quantity. Correspondence establishment section  17  calculates the degrees of similarity between the characteristic quantities of object zones and synthesized characteristic quantities, and finds an optimum correspondence between the objects and object zone, on the basis of calculated degrees of similarity.

TECHNICAL FIELD

The present invention relates to an object-tracking device, anobject-tracking method and an object-tracking program for tracking anobject based on image information, and particularly to anobject-tracking device, an object-tracking method and an object-trackingprogram that enables continued following of a target object even when aplurality of objects become positioned in a superposed relationship on adisplay screen.

BACKGROUND ART

It is required for an object-tracking device, adapted for tracking anobject such as a human body based on image information provided fromtime to time from a video camera, etc., to continue following themovement of the object to be tracked even when a plurality of theobjects become positioned in a crossover or superposed relationship. JPH06-169458 A1 describes an object-tracking device capable of continuingto follow the movements of objects even when the objects intersect whiletracking is being conducted.

FIG. 1 is a block diagram illustrating the construction of anobject-tracking device of prior art described in JP H06-169458 A1. Theobject-tracking device, as shown in FIG. 1, comprises object zoneextracting means 41 for extracting a object zone from the imageinformation periodically provided to input terminal 40 by a video cameraor the like; state-of-following-up detecting means 42 adapted forcomparing the extracted object zone with the object zone to be trackedto detect the state of the object that is to be tracked, wherein theobject zone to be tracked is provided from zone selecting means 45 thatwill be described later; characteristic-quantity generating means 43 forgenerating characteristic quantities of objects based on the imageinformation and the object zone to be tracked; object identifying means44 for generating characteristic quantities of the entire object zonesbased on both the image information and the object zones extracted byobject zone extracting means 41 as well as for selecting the object zonehaving the characteristic quantity closest to the characteristicquantity of the object received from characteristic quantity generatingmeans 43; and zone selecting means 45 to provide as an output the objectzone that is to be tracked.

In the above description, the term “an object zone” means the zone thatincludes an object in the image. The state of an object to be trackedprovided by state-of-following-up detecting means 42 includes: asuperposed or crossover state in which the image of an object to betracked intersects with that of another object; and a tracked state inwhich an object to be tracked is tracked in a stand-alone state.Characteristic quantity generating means 43 has a memory means forstoring the generated characteristic quantities and furthermore includesupdating means for updating the characteristic quantity stored in thememory means. Characteristic quantity generating means 43, when thestate of the object is in the tracked state, updates the characteristicquantity stored in the memory means, while it maintains thecharacteristic quantities stored in the memory means unchanged if thestate of the object is in a superposed state.

Object identifying means 44 selects the object zone that has thecharacteristic quantity most resembling the characteristic quantity ofthe object zone stored in characteristic quantity generating means 43only if the object of interest is in a superposed state. Zone selectingmeans 45 selects, as an object zone to be tracked, the object zoneselected by object identifying means 44 when the state of the objectmakes transition from the state of the superposed state to the trackedstate. Except for the case when the state of the object makes transitionfrom the superposed state to the tracked state, zone-selecting means 45selects, from all the object zones extracted by object zone extractingmeans 41, the object zone present in the position nearest the objectzone that has been previously tracked and provides the selected objectzone as an output. In this way, zone-selecting means 45 provides theselected object zone as a new object zone to be tracked. In the abovedescription, “the object zone that has been previously tracked” meansthe object zone to be tracked that is determined on the basis of theimage information from one frame that precedes the most recent imageinformation received image information from a video camera etc.

In this way, the object-tracking device continuously performs trackingof an object zone through the use of the most recently received imageinformation when the object is in the tracked state. When the state ofthe object makes transition from the tracked state to the superposedstate, the characteristic quantity of the object immediately precedingthe transition is saved. When the state of the object makes transitionfrom the superposed state to the tracked state, the tracking of theobject zone is continuously performed with the object zone selected byobject identifying means 44 that is taken as the object zone to betracked. Accordingly, the tracking of the object zone can be continuedeven when the transition from the tracked state to the superposed stateand subsequent recovery to the tracked state takes place in the state ofthe object.

As described above, the object-tracking device of prior art has beencapable of continuing the tracking of an object zone even in the casewhere a plurality of objects intersect by associating an object with anobject zone, when the object leaves the superposed state, on the basisof the characteristic quantity of the object immediately before theoccurrence of the superposed state.

In addition, the article, Tracking Interacting People, presented by S.J. McKenna, S. Jabri, Z. Duric, and H. Wechsler in Proceedings of theFourth IEEE International Conference on Automatic Face and GestureRecognition, Mar. 28-30, 2000, pp. 348-353 describes a method forcontinuing tracking of an object when intersecting objects part fromeach other to leave the superposed state, in which the probability ofthe object of interest to belong to each of the object zones iscalculated and the object is associated with the object zone of thelargest probability calculated.

The object-tracking device of prior art, however, is liable to lead toan erroneous correspondence between an object and an object zone in thecase where a plurality of objects are contained in a single object zoneat the time when a superposed state is created through intersection of aplurality of objects and then the superposed state is dropped off, or inthe case when objects present in a certain object zone are replaced withother objects before and after creation of the superposed state ofinterest, because the characteristic quantities of the objects justbefore the change in the superposed state accord with none of thecharacteristic quantities of the objects included in the relevant objectzone after the change in the superposed state. For example, in the casewhen four people (A, B, C and D) intersect and then part into groups oftwo people each (a group of A and B and a group of C and D), or in thecase when two people (A and B) and person (C) intersect and then thegrouping of the people changes to create the groups of person (A) andtwo people (B and C), the object-tracking device often mistakescorrespondence between an object and an object zone. Furthermore, whileconventional object-tracking devices select, with each object, theobject zone that have the highest similarity to the object of interest,this way of association between object and object zone does not alwaysguarantee consistent correspondences between overall objects and objectzones.

DISCLOSURE OF THE INVENTION

It is an object of the present invention to provide a device and amethod for tracking an object and also to provide a program for trackingan object that enable correct association between an object and anobject zone even in the case when, after a plurality of objects of asuperposed state in an image have parted from one another, a pluralityof objects are still present in an image zone, or in the case when thegrouping of objects, each group being made up of superposed objects inthe image, changes to counterchange the objects included in the groupsbefore and after a crossover of the groups. It is another object of thepresent invention to enable the selection of optimum correspondencesbetween overall objects and object zones when associating objects andobject zones into corresponding relationship.

The object-tracking device according to the present invention is anobject-tracking device for tracking an object based on imageinformation, comprising: a characteristic-quantity synthesizing meansadapted to synthesize object characteristic quantities representative ofcharacteristic quantities of respective objects included in the imageinformation to generate synthesized characteristic quantities; and acorrespondence-establishing means for establishing correspondencesbetween objects and object zones on the basis of degrees of similaritybetween characteristic quantities of the object zones and thesynthesized characteristic quantities, wherein the object zones refer tothe zones that are extracted from the image information and furtherinclude the objects of interest.

Preferably, the characteristic-quantity synthesizing means synthesizescharacteristic quantities for each of all required combinations of aplurality of objects to generate a synthesized characteristic quantity,and the correspondence-establishing means establishes correspondencesbetween objects and object zones through comparing each of thesynthesized characteristic quantities generated by thecharacteristic-quantity synthesizing means and zone characteristicquantities representative of the characteristic quantities of objectzones. By such construction, an optimum correspondence relation can beopted for between the objects and object zone when establishing thecorrespondence between objects and object zone.

Preferably, the object-tracking device is provided with: an object-zoneextracting means for extracting the object zones from the imageinformation and providing the object-zone information that includes theimage information about the object zones, a state-of-tracking decidingmeans for deciding the states of tracking of individual objects orobject zones, wherein the state-of-tracking means relative positions ofeach object with respect to other objects, and a characteristic-quantitygenerating means for generating the zone characteristic quantities andobject characteristic quantities through the use of the imageinformation, the object-zone information and the decision resultseffected by the state-of-tracking deciding means, wherein thecharacteristic-quantity synthesizing means generates synthesizedcharacteristic quantities through the use of said object characteristicquantities and decision results effected by the state-of-trackingdeciding means. By such construction, the synthesized characteristicquantities can be generated on the basis of relative positions withrespect to other objects.

Preferably, the state-of-tracking deciding means decides the states oftracking of respective objects or object zones based on the object-zoneinformation and the correspondence information that has been determinedthat indicates the corresponding relationship of the object zones andobjects prior to the present to provide the first zone-correspondenceinformation that indicates the corresponding relationship of the objectzones and objects and the states of tracking, thecharacteristic-quantity generating means generates zone characteristicquantities and object characteristic quantities based on the currentimage information, the object-zone information, the firstzone-correspondence information and the correspondence information thathas been determined, the characteristic-quantity synthesizing meansgenerates synthesized characteristic quantities that serve as candidatesto be placed in the corresponding relationship to individual objectzones based on the object characteristic quantities and the firstzone-correspondence information to provide synthesizedcharacteristic-quantity information, wherein the synthesizedcharacteristic-quantity information is the information that includessynthesized characteristic quantities and the corresponding relationshipbetween the synthesized characteristic quantities and objects used forthe generation of the synthesized characteristic quantities, and thecorrespondence-establishing means includes a correspondence-determiningmeans that associates objects and object zones to place in thecorresponding relationship based on the first zone-correspondenceinformation, zone characteristic-quantity information that is theinformation indicative of the zone characteristic quantities and thesynthesized characteristic-quantity information to provide thecorrespondence information that has been determined in the present time.This construction enables preventing establishment of erroneouscorrespondences between objects and object zones even if a plurality ofobjects are still present in an object zone after an object of interestparts or if groups of objects in a superposed or crossover state eachare counter-changed when the two groups intersect.

In the object-tracking device, it is preferred that the state oftracking includes at least one of or a combination of: a stand-alonestate in which only a single object resides in an object zone; acrossover state in which a plurality of objects correspond to a singleobject zone; and a state of parting that is a transient state in which asingle object zone is parting into a plurality of object zones. Thisenables deciding positional relations of an object relative to otherobjects.

Preferably, the characteristic-quantity generating means generates zonecharacteristic quantities, each including at least one of or one ofcombinations of a color histogram, area, image template and colorhistogram normalized with respect to the area, of the object zone, andfinds an object zone corresponding to the object of interest from thefirst zone-correspondence information and generates at least one or oneof combinations of a color histogram, area, image template and colorhistogram normalized with respect to the area of the object zone as anobject characteristic quantity.

Preferably, the state-of-tracking deciding means includes an object-zonestoring means for storing the object-zone information, anobject-tracking means for tracking an object based on the object-zoneinformation, the correspondence information that has been determined andthe object-zone information prior to the present provided from theobject-zone storing means and further providing a secondzone-correspondence information that indicates the correspondencesbetween objects and object zones, and a state-deciding means fordeciding the states of tracking of objects based on the secondzone-correspondence information, the object-zone information and theobject-zone information prior to the present and providing the firstzone-correspondence information.

Preferably, the state-deciding means, based on at least one of or one ofthe combinations of the correspondences between objects and objectzones, distances between object zones and continued periods ofseparation of the object zones, obtained from the secondzone-correspondence information and object-zone information, groups theobjects that have a common region in their corresponding object zones tosort the objects and corresponding object zones into one class, andsorts the object, which differs in the corresponding object zone fromany other objects, and the object zone corresponding thereto into oneclass to sort the objects and object zones into a plurality of classes,and decides the state of tracking on the basis of the sorted classes.

Preferably, the state of tracking includes the state of parting that isa transient state, through which an object zone parts into a pluralityof object zones,

-   -   the state-deciding means decides that, if two or more object        zones are included in a sorted class, then the class meets the        condition of being in a state of parting, and that, if a class        meets the condition of being in a state of parting, the states        of tracking of the objects and object zones included in the        class are the state of parting.

It is preferred that, if the sorted class meets the condition of beingin the state of parting and if the sorted class meets at least one of orone of the combinations of the conditions that two or more objects areincluded in the class, that each of the distances between the objectzones included in the class exceeds a predetermined threshold and thatcontinued periods of separation of the object zones included in theclass exceed a predetermined threshold, the state-deciding means decidesthat the states of tracking of the objects and object zones included inthe class are the state of parting. This construction enables preventingan erroneous decision to be led that the positional relation of anobject of interest relative to another object is in a state of parting.

Preferably, the state of tracking includes a state of parting and astand-alone state in which a single object resides in an object zone,and if the sorted class includes only one object and if the states oftracking of the object and the object zone included in the class are notthe state of parting, then the state-deciding means decides that thestates of tracking of the object and the object zone included in theclass are the stand-alone state.

It is preferred that the state of tracking includes a state of partingand also a crossover state in which a plurality of objects are incorresponding relationship to a single object zone, and if a sortedclass includes two or more objects and if the states of tracking of theobjects and the object zones included in the class are not the state ofparting, the state-deciding means decides that the states of tracking ofthe objects and the object zones included in the class are the crossoverstate.

Preferably, the characteristic-quantity generating means includes: acharacteristic-quantity extracting means for extracting zonecharacteristic quantities from the image information, object-zoneinformation and the first zone-correspondence information and providingthe zone characteristic-quantity information that is the informationindicative of the zone characteristic quantities;characteristic-quantity storing means for storing object characteristicquantities and selecting the stored object characteristic quantities tosupply the selected object characteristic quantities, as required; andcharacteristic-quantity updating means for updating the objectcharacteristic quantities stored in the characteristic-quantity storingmeans based on the zone characteristic-quantity information, the firstzone-correspondence information or correspondence information that hasbeen determined and the object characteristic quantities generated priorto the present.

It is preferred that the state of tracking includes the state of partingthat is a transient state through which an object zone parts into aplurality of object zones, and the characteristic-quantity extractingmeans includes, into zone characteristic-quantity information, theinformation indicating that there is no need of establishingcorrespondences to objects for the object zones that represent thestates other than the state of parting while in their tracking states,and the correspondence-determining means excludes, from theestablishment of the corresponding relationship, the object zonesindicated in the zone characteristic-quantity information as there is noneed of establishing corresponding relationship to objects. Thisconstruction allows the amount of calculation for calculation of thedegrees of similarity to be reduced.

Preferably, the state of tracking includes a stand-alone state in whicha single object resides in an object zone, and thecharacteristic-quantity updating means decides whether or not the stateof tracking of an object is the stand-alone state on the basis of thefirst zone-correspondence information or the correspondence informationthat has been determined, and if the state of tracking of the object isthe states other than the stand-alone state, does not update the objectcharacteristic quantities stored in the characteristic-quantity storingmeans.

Preferably, the characteristic-quantity synthesizing means determinesall possible combinations of objects and object zones based on theobject characteristic quantities generated by thecharacteristic-quantity generating means and the firstzone-correspondence information, and synthesizes object characteristicquantities only for the determined combinations of objects and objectzones to generate synthesized characteristic quantities. Thisconstruction allows the procedures for generating unnecessarysynthesized characteristic quantities to be deleted.

Preferably, characteristic-quantity synthesizing means calculates thesynthesis ratios that are coefficients for adjusting the ratios at whichthe object characteristic quantities are synthesized, and generatessynthesized characteristic quantities on the basis of the synthesisratios and object characteristic quantities. By the constructiondescribed above, it is enabled to correct an error arising from themagnitude of an object when the magnitude of the object in the imagespace differs from the actual magnitude of the object.

It is preferred that the characteristic-quantity synthesizing meansreceives zone characteristic quantities as well as object characteristicquantities from the characteristic-quantity generating means, calculatessynthesized characteristic quantities depending on desired synthesisratios on the basis of the received zone characteristic-quantityinformation and object characteristic quantities, and provides thesynthesized characteristic quantity for the synthesis ratio that yieldsthe highest of all the degrees of similarity between the calculatedsynthesized characteristic quantities and the zone characteristicquantities. By this construction, it suffices that the degrees ofsimilarity are calculated based on only one synthesized characteristicquantity for all combinations of objects and object zones, wherebyredundant calculation procedures can be omitted.

Preferably, the state of tracking includes a state of parting that is atransient state through which an object zone parts into a plurality ofobject zones, and the characteristic-quantity synthesizing meansgenerates synthesized characteristic quantities only for the objectzones that are indicated as having the state of parting as their statesof tracking. This construction allows the procedures for generatingunnecessary synthesized characteristic quantities to be deleted.

Preferably, the object characteristic quantity includes an area of anobject, and the characteristic-quantity synthesizing means calculatesthe synthesis ratios that are coefficients for adjusting the ratios atwhich the object characteristic quantities are synthesized on the basisof the areas of objects included in the object characteristic quantitiesand generates synthesized characteristic quantities from the synthesisratios and the object characteristic quantities. This constructionallows reducing the amount of calculation for calculating the synthesisratio.

Preferably, the characteristic-quantity synthesizing means limits thesynthesis ratios within a predetermined range on the basis of thevariations in the areas of objects. This construction allows a correctsynthesis ratio to be obtained even when an object area varies in animage.

Preferably, the characteristic-quantity synthesizing means receives zonecharacteristic quantities together with object characteristic quantitiesfrom the characteristic-quantity generating means, calculatessynthesized characteristic quantities within the range of the variationsin the areas of objects based on the received zone characteristicquantities and object characteristic quantities, and provides thesynthesized characteristic quantities that have the highest degrees ofsimilarity to the zone characteristic quantities of the object zones ofinterest. By this construction, it suffices to provide only onesynthesized characteristic quantity for each of object combinations,whereby redundant procedures can be deleted.

It is preferred that the object characteristic quantity includes animage template representative of the shape and color of an object, andthe characteristic-quantity synthesizing means decides the back-to-bellyrelation of each of the objects from the image templates and zonecharacteristic quantities and obtains the synthesized characteristicquantities by synthesizing the image templates based on the respectivedecided back-to-belly relations of the objects. This construction allowsthe correct calculation of the corresponding relationship betweenobjects and object zones even when any object partially overlaps behindanother object.

It is preferred that the correspondence-determining means is providedwith a correspondence-calculating means for calculating the combinationof objects and object zones of the highest similarity from all thepossible combinations of the objects and object zones that are possiblyassociated in corresponding relationship based of the synthesizedcharacteristic-quantity information, the zone characteristic-quantityinformation and the first zone-correspondence information, selecting thecalculated combination of objects and object zones as an optimumcombination and generating the optimum-correspondence information thatindicates the optimum corresponding relationship between objects andobject zones, and a correspondence-deciding means for determining thecorresponding relationship between objects and object zones on the basisof the first zone-correspondence information and theoptimum-correspondence information and providing the correspondenceinformation that has been determined that is the information includingthe decided corresponding relationship between objects and object zones.This construction allows the selection of optimum correspondingrelationship when objects and object zones are associated incorresponding relationship.

Preferably, the correspondence-calculating means calculates a totaldegree of similarity for each of all the possible combinations ofobjects and object zones, the total degree of similarity being a sum ofthe degrees of similarity between the characteristic quantities ofobject zones and synthesized characteristic quantities within eachcombination, and decides to be the combination of the highest similarityon the combination of the highest total degree of similarity, of the allthe possible combinations.

It is preferred that the first zone-correspondence information includesthe information about an at-rest/in-motion state that indicates whetheran object zone is at rest or in motion, and thecorrespondence-calculating means excludes the combination of the objectand object zone that is indicated as being at rest in the informationabout an at-rest/in-motion state from all the possible combinations.This construction allows the amount of calculation of thecorrespondence-calculating means to be reduced by calculatingexclusively the combinations in which the object zones that exhibit thestate of at-rest in their at-rest/in motion states cannot be associatedin corresponding relationship to any objects.

It is preferred that if the degrees of combined similarity that can beobtained from the degrees of similarity of the sets of the objects andobject zones, the sets of the objects and object zones making up thecombinations decided to be ranked as the highest similarity, are equalto or lower than a predetermined threshold, then thecorrespondence-calculating means selects, from the combinations of thedegrees of combined similarity ranked as the highest similarity of allpossible combinations of objects and object zones, the combinations ofthe degrees of combined similarity within the predetermined threshold,includes the corresponding relationship of objects and object zonescommon to the selected combinations, into the optimum-correspondenceinformation as optimum correspondences, and further, for the objects andobject zones having the corresponding relationship that are not includedin the corresponding relationship of the object and object zone commonto the selected combinations, includes the information indicating thatthere are no optimum correspondence between the objects and objectzones, into the optimum-correspondence information; for the objects notindicated as having no optimum corresponding relationship to any objectzones in the optimum-correspondence information, thecorrespondence-deciding means provides the information indicating thecorresponding relationship of objects and object zones included in theoptimum-correspondence information as the correspondence informationthat has been determined; and for the objects indicated as having nooptimum corresponding relationship to any object zones in theoptimum-correspondence information, the correspondence-deciding meansprovides the information indicating the corresponding relationship ofobjects and object zones included in the first zone-correspondenceinformation as the correspondence information that has been determined.This construction allows the selection of erroneous correspondingrelationship between objects and object zones to be obviated.

Preferably, the state of tracking includes a state of parting that is atransient state through which an object zone parts into a plurality ofobject zones, and the correspondence-deciding means determines thecorresponding relationship between objects and object zones to beindicated in the optimum-correspondence information only for the objectzones that exhibit a state of parting as their states of tracking. Thisconstruction allows reduction of the procedures for deciding thecorresponding relationship between objects and object zones.

Preferably, the state of tracking includes a state of parting that is atransient state through which an object zone parts into a plurality ofobject zones, and the correspondence-deciding means provides thecorrespondences between objects and object zones included in the firstzone-correspondence information as the correspondence information thathas been determined only for the object zones that exhibit states otherthan the state of parting while in their tracking states. Thisconstruction allows reduction of the procedures for deciding thecorresponding relationship between objects and object zones.

The object-tracking method according to the present invention is anobject-tracking method intended for tracking an object based on imageinformation. The method synthesizes object characteristic quantities,which represent characteristic quantities of respective objects includedin the image information, to generate a synthesized characteristicquantity, and establishes a correspondence between object or objects andobject zone on the basis of the degree of similarity between thesynthesized characteristic quantity and characteristic quantity of theobject zone, wherein the object zone is a region extracted from theimage information and including the object or objects.

The object-tracking method preferably synthesizes characteristicquantities for each of all required combination of a plurality ofobjects to generate a synthesized characteristic quantity, andestablishes corresponding relationship between object or objects andobject zone through comparison of the generated synthesizedcharacteristic quantity and zone characteristic quantity that representsthe characteristic quantity of the object zone. The above-describedmethod enables the selection of optimum corresponding relationshipbetween objects and object zones when objects and object zones are to beassociated in corresponding relationship.

The object-tracking method preferably extracts an object zone from theimage information to provide object-zone information that includes theimage information about the object zone, decides a state of trackingrepresentative of a relative position with respect to another object forevery object or object zone, generates zone characteristic quantities,object characteristic quantities through the use of the imageinformation, the object-zone information, and the decision results, andgenerates synthesized characteristic quantities through the use of theobject characteristic quantities and the decision results. Theabove-described method enables generation of synthesized characteristicquantities based on the positional relations relative to other objects.

It is preferred that the object-tracking method decides the state oftracking of every object or every object zone based on the object-zoneinformation and the correspondence information that has been determinedthat indicates the corresponding relationship between object zones andobjects prior to the present to provide the first zone-correspondenceinformation indicative of the corresponding relationship between theobjects and object zones and the states of tracking; generates zonecharacteristic quantities and object characteristic quantities based onthe present image information, the object-zone information, the firstzone-correspondence information and the correspondence information thathas been determined; generates a synthesized characteristic quantitythat functions as candidate to be placed in corresponding relationshipto each object zone on the basis of the object characteristic quantitiesand the first zone-correspondence information to provide the synthesizedcharacteristic-quantity information, which is the information includingthe synthesized characteristic quantities and the correspondingrelationship between synthesized characteristic quantities and objectsused for generating the synthesized characteristic quantities ofinterest; and establishes correspondences between objects and objectzones based on the first zone-correspondence information, zonecharacteristic-quantity information, which is the information indicativeof the zone characteristic quantities, and the synthesizedcharacteristic-quantity information, to provide the correspondenceinformation that has been determined at present. The above-describedmethod enables preventing erroneous correspondences between objects andobject zones at the event that a plurality of objects are present in anobject zones after an object parts or that objects counterchange whenintersection of the objects takes place.

In the object-tracking method, the state of tracking includes astand-alone state, in which only a single object is present in an objectzone, a crossover state, in which a plurality of objects are present ina single object zone, or a state of parting that is a transient statethrough which an object zone parts into a plurality of object zones.

The object-tracking method is preferably developed such that the methodgenerates, as a zone characteristic quantity, at least one of, or one ofthe combinations of the color histograms, areas, image templates andcolor histograms normalized with respect to respective areas, of objectzones, and seeks the object zones corresponding to the objects from thefirst zone-correspondence information, and generating, as an objectcharacteristic quantity, at least one of, or one of the combinations ofthe color histograms, areas, image templates and color histogramsnormalized with respect to respective areas of the object zones.

The object-tracking method is preferably developed such that the methodstores the object-zone information, tracks an object on the basis of theobject-zone information, the correspondence information that has beendetermined and the object-zone information prior to the present toprovide a second zone-correspondence information indicative of thecorrespondence between the object and object zone, and decides the stateof tracking an object on the basis of the second zone-correspondenceinformation, the object-zone information and the object-zone informationprior to the present to provide the first zone-correspondenceinformation.

The object-tracking method is preferably developed such that the methodsorts objects and object zones into a plurality of classes by: groupingthe objects, to which the object zones having a common regioncorrespond, to enrol the objects and the corresponding object zones inone class; and for the objects that correspond to the object zonesdiffering from the object zones corresponding to any other objects,enrolling the objects and the corresponding object zone in one class,based on at least one of, or one of combinations of the correspondingrelationship between objects and object zones, the distances betweenobject zones, and the duration period for on-parting of object zonescalculated from the second zone-correspondence information and theobject-zone information; and decides the state of tracking based on theclassified class.

The object-tracking method is preferably developed such that the stateof tracking includes a state of parting that is a transient statethrough which a single object zone parts into a plurality of objectzones, and the object-tracking method decides that a classified classmeets the condition of being in the state of parting if the classincludes two or more object zones, and if a class meets the condition ofbeing in the state of parting, decides to be the state of parting on thestates of tracking of the objects and object zones included in the classof interest.

The object-tracking method is preferably developed such that if theclassified class meets the condition of being in the state of partingand further meets at least one of, or one combination of the conditionsthat the class includes two or more objects, that the distances betweenthe object zones included in the class exceed a predetermined threshold,and that the continued periods of separation of the object zonesincluded in the class exceed a predetermined threshold, theobject-tracking method decides to be a state of parting on the states oftracking of the objects and object zones included in the class. Theabove-described method enables preventing the positions of an objectrelative to other objects from being erroneously decided to be in thestate of parting.

The object-tracking method is preferably developed such that the stateof tracking includes the state of parting and a stand-alone state inwhich a single object is present in an object zone, and theobject-tracking method decides that the states of tracking of the objectand object zone are the stand-alone state if the classified classincludes a single object and also neither of the states of tracking ofthe object and the object zone included in the class is the state ofparting.

The object-tracking method is preferably developed such that the stateof tracking includes the state of parting and the crossover state inwhich a plurality of objects correspond to a single object zone, and theobject-tracking method decides that the states of tracking of theobjects and object zone included in the classified class are thecrossover state if the class includes two or more objects and neither ofthe states of tracking of the objects and the object zone included inthe class is the state of parting.

The object-tracking method is preferably developed such that the methodextracts zone characteristic quantities from the image information, theobject-zone information and the first zone-correspondence information toprovide the zone characteristic-quantity information, which is theinformation indicative of the zone characteristic quantities, stores theobject characteristic quantities, selects the stored objectcharacteristic quantities to be supplied as required, and updates thestored object characteristic quantities on the basis of the zonecharacteristic-quantity information, the first zone-correspondenceinformation or the correspondence information that has been determinedand the object characteristic quantities generated prior to the present.

The object-tracking method is preferably developed such that the stateof tracking includes the state of parting that is a transient statethrough which an object zone parts into a plurality of object zones, andthe object-tracking method includes, into the zonecharacteristic-quantity information, the information which indicatesthat there is no need of establishing the corresponding relationship toany objects, for the object zones that are indicated as having thestates other than the state of parting while in their tracking states,and excludes the object zones, which are indicated in thezone-correspondence information that there is no need of establishingthe corresponding relationship to any objects, from the establishment ofcorresponding relationship. The method described above allows reducingthe amount of calculation for calculating the degrees of similarity.

The object-tracking method is preferably developed such that the stateof tracking includes a stand-alone state in which a single object ispresent in an object zone, and the object-tracking method decideswhether or not the state of tracking is the stand-alone state based onthe first zone-correspondence information or correspondence informationthat has been determined, and skips an update of the stored objectcharacteristic quantity if the state of tracking of an object is any ofthe states other than the stand-alone state. The above-describedprocedures allow deletion of update processes of unnecessary objectcharacteristic quantities.

The object-tracking method is preferably developed such that the methoddetermines all possible combinations of objects and object zones on thebasis of the object characteristic quantities and the firstzone-correspondence information, and synthesizes object characteristicquantities to generate synthesized characteristic quantities only forthe determined combinations of objects and object zones. In this way,the processes of generating unnecessary synthesized characteristicquantities can be deleted.

The object-tracking method is preferably developed such that the methodfinds synthesis ratios, which are the coefficients for adjusting theratios of the object characteristic quantities to be synthesized, andgenerates synthesized characteristic quantities on the basis of thesynthesis ratios and object characteristic quantities. By the proceduresdescribed above, it is enabled to correct an error arising from themagnitude of an object when the magnitude of the object in the imagespace differs from the actual magnitude of the object.

The object-tracking method is preferably developed such that the methodreceives zone characteristic quantities together with objectcharacteristic quantities, calculates synthesized characteristicquantities for arbitrary synthesis ratios based on the received zonecharacteristic-quantity information and the object characteristicquantities, and provides the synthesized characteristic quantitycorresponding to the highest degree of similarity between the zonecharacteristic quantity and the calculated synthesized characteristicquantity. Through implementing the above-described procedures, basing ononly one synthesized characteristic quantity for each combination ofobjects and object zone suffices for calculation of a degree ofsimilarity, whereby redundant calculation procedures can be omitted.

The object-tracking method is preferably developed such that the stateof tracking includes a state of parting, the state of parting being atransient state through which an object zone parts into a plurality ofobject zones, and the object-tracking method generates synthesizedcharacteristic quantities only for the object zones indicated as beingin the state of parting while in their tracking states. Theabove-described procedures allow reducing the procedures for generatingredundant synthesized characteristic quantities.

The object-tracking method is preferably developed such that the objectcharacteristic quantity includes an area of an object, and theobject-tracking method calculates a synthesis ratio, which is acoefficient for adjusting the ratios of the object characteristicquantities to be synthesized, on the basis of the areas of objects andgenerating a synthesized characteristic quantity from the calculatedsynthesis ratio and object characteristic quantities. Theabove-described procedures reduce the amount of calculation forcalculating a synthesis ratio.

The object-tracking method is preferably developed such that thesynthesis ratio is restricted within a range predetermined on the basisof the variations of the object areas. The above-described procedureenables obtaining a correct synthesis ratio even when the area of anobject varies in the image.

The object-tracking method is preferably developed such that the methodreceives the zone characteristic quantities together with the objectcharacteristic quantities, generates synthesized characteristicquantities within the range of variations in the object areas based ofthe received zone characteristic quantities and object characteristicquantities, and provides the synthesized characteristic quantity of thehighest degree of similarity to the zone characteristic quantity of theobject zone of interest. The above-described procedures need provideonly one synthesized characteristic quantity for each of thecombinations of objects, thereby enabling redundant processing to bedeleted.

The object-tracking method is preferably developed such that the objectcharacteristic quantity includes an image template, which describes ashape and/or color of an object, and the object-tracking method decidesthe back-to-belly relations of the objects on the basis of the imagetemplates and zone characteristic quantities, and synthesizes the imagetemplates based on the decided back-to-belly relations of the objects toobtain a synthesized characteristic quantity. The above-describedprocedures enable correct calculation of the correspondence relationbetween an object and an object zone even when the object is partiallyhidden behind another object.

The object-tracking method is preferably developed such that the methodcalculates, based on the synthesized characteristic-quantityinformation, the zone characteristic-quantity information and the firstzone-correspondence information, the combination of objects and objectzones of the highest degree of similarity from all the possiblecombinations of objects and object zones that can be associated incorresponding relationship, selects the calculated combination ofobjects and object zones as the objects and object zone of an optimumcorrespondence, and generates the optimum-correspondence informationthat indicates the optimum correspondence relation of the objects andobject zone; and determines corresponding relationship between objectsand object zones based on the first zone-correspondence information andthe optimum-correspondence information and provides the correspondenceinformation that has been determined, which is the information inclusiveof the determined corresponding relationship between objects and objectzones.

The object-tracking method is preferably developed such that the methodcalculates a total degree of similarity for all possible combinations ofobjects and object zones, wherein the total degree of similarity is asum of the degrees of similarity between characteristic quantities ofobject zones and synthesized characteristic quantities in each of thecombinations, and decides that the combination of the highest totaldegree of similarity of all the possible combinations is the combinationof the highest similarity.

The object-tracking method is preferably developed such that the firstzone-correspondence information includes the information about anat-rest/in-motion state that indicates whether an object zone is at restor in motion, and the object-tracking method excludes, from all thepossible combinations of the objects and object zones, the combinationof object and object zone indicated as being in the at-rest state by theinformation about an at-rest/in-motion state. According to theabove-described method, it suffices only to calculate the combinationsin which the object zones that exhibit the at-rest state in theirat-rest/in-motion states do not correspond to any objects, whereby theamount of calculation is reduced.

The object-tracking method is preferably developed such that, if thedegrees of combined similarity, obtained from the degrees of similarityof the sets of objects and an object zones that make up the combinationdecided as a combination of the highest similarity, is equal to or lowerthan a predetermined threshold, then the object-tracking method selectsthe combinations of the degrees of combined similarity within apredetermined threshold from the combinations of the degrees of combinedsimilarity ranked as the highest similarity, of all the possiblecombinations of objects and object zones, includes the correspondingrelationship between objects and object zones common to the selectedcombinations into the optimum-correspondence information as optimumcorrespondences, and further includes the information that indicatesabsence of an optimum correspondence between any object and object zoneinto the optimum-correspondence information, for the object and objectzone in the correspondence relation that is not included in thecorresponding relationship of the objects and object zones common to theselected combinations; for the objects not indicated in theoptimum-correspondence information that are absent from the optimumcorresponding relationship to object zones, provides the informationindicating the corresponding relationship between the objects ofinterest and object zones included in the optimum-correspondenceinformation, as the correspondence information that has been determined;and for the objects indicated in the optimum-correspondence informationthat are absent from the optimum corresponding relationship to objectzones, provides the information indicating the correspondingrelationship between the objects of interest and object zones includedin the first zone-correspondence information, as the correspondenceinformation that has been determined. According to the above-describedprocess, it is enabled to obviate selection of erroneous correspondencesbetween objects and object zones.

The object-tracking method is preferably developed such that theobject-tracking method determines the corresponding relationship betweenobjects and object zones to be identical with those indicated in theoptimum-correspondence information only for the object zones indicatedas having the state of parting while in their tracking states. Theabove-described procedure deletes the processing of determining thecorresponding relationship between objects and object zones.

The object-tracking method is preferably developed such that theobject-tracking method provides the correspondences between objects andobject zones included in the first zone-correspondence information asthe correspondence information that has been determined only for theobject zones indicated as having the state other than the state ofparting with their states of tracking. The above-described proceduredeletes the processing of determining the corresponding relationshipbetween objects and object zones.

The object-tracking program according to the present invention is anobject-tracking program for tracking an object based on imageinformation, the program operating a computer to execute the processescharacterized by steps of receiving image information, synthesizingobject characteristic quantities that represent characteristicquantities of respective objects included in the received imageinformation and generating a synthesized characteristic quantity; andestablishing a correspondence between the objects and an object zonebased on the degree of similarity between a characteristic quantity ofthe object zone and the synthesized characteristic quantity, the objectzone being a region that is extracted from the image information andalso includes the objects.

According to another aspect of the present invention, theobject-tracking program is an object-tracking program that establishescorrespondences between objects and object zones included in receivedimage information, and the program is developed to operate a computer toexecute processes of: receiving image information; extracting objectzones from the received image information and providing the object-zoneinformation inclusive of image information about the object zones;deciding the state of tracking with each object or object zone on thebasis of the object-zone information and the correspondence informationthat has been determined indicating the corresponding relationship ofthe objects and object zones prior to the present and providing firstzone-correspondence information, which indicates the correspondingrelationship of the object zones and objects and the states of tracking;generating the zone characteristic quantities, which represent thecharacteristic quantities of the object zones, and the objectcharacteristic quantities, which represent the characteristic quantitiesof the objects through the use of the image information, the object-zoneinformation and the first zone-correspondence information; synthesizingcharacteristic quantities for all required combinations of a pluralityof objects to generate each of synthesized characteristic quantitiesbased on the object characteristic quantities and the firstzone-correspondence information, and providing synthesizedcharacteristic-quantity information, which is the information includingthe synthesized characteristic quantities and the correspondingrelationship between the objects used for generating synthesizedcharacteristic quantities and the synthesized characteristic quantities;and associating the objects and object zones in correspondingrelationship based on the first zone-correspondence information, thezone characteristic-quantity information and the synthesizedcharacteristic-quantity information, and providing the correspondenceinformation that has been determined for the present.

According to a further aspect of the present invention, theobject-tracking program is an object-tracking program that establishescorrespondences between objects and object zones included in receivedimage information, and the program is developed to operate a computer toexecute processes of: receiving image information; deciding the state oftracking with each object or object zone on the basis of the object-zoneinformation and the correspondence information that has been determinedindicating the corresponding relationship of the objects and objectzones prior to the present and providing first zone-correspondenceinformation, which indicates the corresponding relationship of theobject zones and objects and the states of tracking; generating the zonecharacteristic quantities, which represent the characteristic quantitiesof the object zones, and the object characteristic quantities, whichrepresent the characteristic quantities of the objects through the useof the image information, the object-zone information and the firstzone-correspondence information; and while taking each of objects as atarget, deciding the correspondences between the objects and objectzones through designating the correspondences between objects and objectzones included in the first zone-correspondence information as thecorrespondences between the objects and object zones, for the objectsdecided to be in the states other than a state of parting while in theirtracking states, the state of parting being a transient state throughwhich an object zone parts into a plurality of object zones; for theobjects decided to be in the state of parting while in their trackingstates, synthesizing characteristic quantities for all requiredcombinations of a plurality of objects on the basis of the objectcharacteristic quantities and the first zone-correspondence informationto generate respective synthesized characteristic quantities; andcomparing each of synthesized characteristic quantities with a zonecharacteristic quantity to associate the objects corresponding to thesynthesized characteristic quantity, which has the highest degree ofsimilarity to an object zone, with the object zone of interest to placein corresponding relationship.

The present invention offers the following effects:

According to the present invention, the procedures of synthesizingcharacteristic quantity of a plurality of objects and calculating thedegree of similarity between the objects and an object zone by the useof synthesized synthesizing characteristic quantity enables erroneousestablishment of the correspondence relation between objects and objectzones to be prevented even when a plurality of objects are present in anobject zone after parting of an object takes place or groups of objectsare positionally counterchanged when the groups intersect. Inparticular, in the correspondence-establishment of objects and objectzones when a crossover state is dissolved, the present invention enableserroneous establishment of the corresponding relationship betweenobjects and object zones to be prevented; and

-   -   allowing for combinations of objects and object zones and        calculating a degree of combined similarity for each of the        combinations allows overall optimum correspondences to be opted        for when objects and object zones are associated in        corresponding relationship.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating the construction of aconventional object-tracking device;

FIG. 2 is a block diagram illustrating an example of a construction ofthe object-tracking device of the present invention;

FIG. 3 is a block diagram illustrating an example of a construction ofthe correspondence establishment section;

FIG. 4 is a flow chart representing an example of the process of theobject-tracking device according to the present invention;

FIG. 5 is a schematic diagram representing an example of the firstzone-correspondence information;

FIG. 6 is a block diagram representing an example of the construction ofthe state-of-tracking decision section;

FIG. 7 is a schematic diagram representing an example of the secondzone-correspondence information;

FIG. 8 is a block diagram representing an example of the construction ofthe characteristic-quantity generation section;

FIG. 9 is a block diagram representing an example of the construction ofthe correspondence decision section;

FIG. 10 is a schematic diagram representing an example of a method ofsynthesizing characteristic quantities through the use of templates;

FIG. 11 is a block diagram representing another example of theconstruction of the object-tracking device according to the presentinvention;

FIG. 12 is a block diagram representing a further example of theconstruction of the object-tracking device according to the presentinvention;

FIG. 13 is a block diagram representing an example of the constructionof the first control section; and

FIG. 14 is a flow chart representing another example of the process ofthe object-tracking device according to the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION

(The First Working Example of the Invention)

Explanation is next presented with reference to drawings regarding afirst embodiment of the present invention. FIG. 2 is a block diagramillustrating an example of the construction of the object-trackingdevice according to the present invention. The object-tracking device,as shown in FIG. 2, includes: image input terminal 11 that receivesimage information ever incoming from a video camera or the like;object-zone extraction section 12 that extracts an object zone from theimage information supplied to image input terminal 11;characteristic-quantity generation section 14 that creates both acharacteristic quantity of an object and a characteristic quantity of anobject zone based on the image information supplied to image inputterminal 11; characteristic-quantity synthesis section 15 thatsynthesizes characteristic quantities of a plurality of objects tocreate a new characteristic quantity; and correspondence establishmentsection 17 that establishes correspondence between an object and anobject zone.

In addition, although object-zone extraction section 12,characteristic-quantity generation section 14, characteristic-quantitysynthesis section 15 and correspondence establishment section 17 can berealized through hardware, they can be realized through software aswell. In other words, object-zone extraction section 12,characteristic-quantity generation section 14, characteristic-quantitysynthesis section 15 and correspondence establishment section 17 can berealized by means of the programs stored in a memory to perform theirfunctions described below, and also a CPU that executes procedures inaccordance with the programs.

The object-tracking device tracks a material body by continuously takingpictures within a prescribed scope (hereinafter referred to as atracking scope) through the use of a video camera or the like andrecognizing the images of the material body that come out in the takenpictures.

For example, a video camera is immovably fixed and continuously takespictures within a predetermined scope. In addition, the video camera canbe varied in the photo-shooting scope vertically or in the left-to-rightdirection. Furthermore, an alternative imaging device capable of takingimages at determined time periods can be employed in place of continuousimage-scanning through the use of a video camera. It is also feasible toemploy a device for acquiring image information in the form of staticimages, such as a digital camera and the like.

An object zone denotes the region in the image that includes an object,wherein an object refers to an image of a material body. The object zoneis extracted, for example, by the following steps: acquiring of imagesof the tracking scope in absence of the object is effected by means of avideo camera or the like; object-zone extraction section 12 stores theacquired image information in a background-image storing device asbackground image information; object-zone extraction section 12 comparesthe contents of the image information received by image input terminal11 and background image information stored in the background-imagememory in pixel units to calculate the differences of the pixel valuesand extracts the pixels that have differences in excess of apredetermined threshold (hereinafter the pixels are referred to asobject pixels); object-zone extraction section 12 selects all contiguousobject pixels from the extracted pixels; object-zone extraction section12 interconnects the selected object pixels to extract zones; andobject-zone extraction section 12 labels zone numbers to the extractedzones and supplies as object zones.

In addition, background-image storing device can store not only a singlecontent of the background image information but also a plurality ofcontents of the background image information. For example, if thetracking scope is outdoors, background-image storing device may storeboth the background image information acquired during daylight(hereinafter referred to as a daylight image) and that acquired at night(hereinafter referred to as a night image) to allow employing thedaylight image in daylight and the night image at night.

If the characteristic quantity, such as shape or the like, of the targetobject is known, the object-tracking device may be provided with anobject-model storing device adapted to store in advance the informationabout the shape, color, operation or the like of the object asobject-model information, in lieu of the background-image storingdevice. In this case, object-zone extraction section 12 can collate thereceived image information with the object model that has been stored inthe object-model storing device and extract the zone that matches withthe object model as an object zone.

A characteristic quantity is a quantity that serves for establishing acorrespondence between an object and an object zone. The characteristicquantity involves two kinds of the quantities, e.g., a characteristicquantity of an object zone (referred to as an object-zone characteristicquantity or a zone characteristic quantity) and a characteristicquantity of an object (referred to as an object characteristicquantity): the zone characteristic quantity represents a characteristicquantity acquired from each of object zones and the objectcharacteristic quantity represents characteristic quantities acquiredfrom an assembly of object zones that constitute an object.

Characteristic-quantity generation section 14 calculates, for example, acolor histogram of an object zone, and supplies the calculated colorhistogram of the object zone as a zone characteristic quantity.Alternatively, the characteristic quantity can be an area of the objectzone, an image template or the like, or it can be a color histogramnormalized with respect to an area. Further, it can be a combination oftwo or more characteristic quantities selected from the group of a colorhistogram, an area, an image template and a color histogram normalizedwith respect to an area.

The characteristic quantity of an object as well can be represented byany of the color histogram, the area, the image template and the colorhistogram normalized with respect to an area or a combination of them,because the characteristic quantity of an object is a characteristicquantity composed of an assembly of the characteristic quantities of theobject zones that constitutes the object.

Characteristic-quantity generation section 14 calculates thecharacteristic quantity of an object on the basis of the characteristicquantities of the object zones. The characteristic quantity of an objectzone is a characteristic quantity calculated from each object zone, andthe characteristic quantity of an object is a characteristic quantitycalculated from an assembly of the object zones that constitute theobject. Accordingly, if an object corresponds to a single object zone,the characteristic quantity of the object is determined on the basis ofthe characteristic quantity of the single object zone and if an objectis constituted by a plurality of object zones, the characteristicquantity of the object is determined from the synthesis of thecharacteristic quantities of the object zones.

It is remarked that object-zone extraction section 12 is a specificexample (an embodiment) of an object-zone extracting means adapted forextracting an object zone from image information and providing theobject zone information that includes the image information about theextracted object zone. Further, characteristic-quantity generationsection 14 is a specific example of the characteristic-quantitygenerating means adapted for generating a characteristic quantity of theobject zone and a characteristic quantity of the object, andcharacteristic-quantity synthesis section 15 is a specific example ofcharacteristic-quantity synthesizing means adapted for generatingsynthesized characteristic quantities by synthesizing the characteristicquantities for all required combinations of a plurality of objects.

FIG. 3 is a block diagram illustrating an example of construction ofcorrespondence establishment section 17. Correspondence establishmentsection 17 has state-of-tracking decision section 13 and correspondencedecision section 16, as shown in FIG. 3. State-of tracking decisionsection 13 establishes the correspondence between an object and anobject zone and decide the state of tracking. Correspondence decisionsection 16 confirms the correspondence between the object and objectzone on the basis of the state of tracking of the object.

In the above description, the state of tracking refers to the state thatrepresents the relative position of an object with respect to anotherobject such as crossover with and parting from another object. The stateof tracking involves, for example, a stand-alone state in which a singleobject is present in an object zone, a crossover state in which aplurality of objects correspond to a single object zone, and a state ofparting or an on-parting state, i.e. a transient state in which anobject zone is just on parting into a plurality of object zones. Inother words, the state of tracking of the present embodiment includes astand-alone state, a crossover state and a state of parting. Thecrossover state generally makes transition to the stand-alone state byway of the state of parting.

In addition, state-of-tracking decision section 13 is a specifiedexample of a state-of-tracking deciding means adapted to decide the sateof tracking with each object or object zone wherein the state oftracking represents a positional relation relative to another object.Correspondence decision section 16 is a specific example of thecorrespondence establishing means, which is adapted to establish thecorrespondence between an object and an object zone on the basis of thedegree of similarity between a synthesized characteristic quantity andthe characteristic quantity of the object zone, which is the zonecontaining the objects, and also correspondence deciding means, which isadapted to supply the currently correspondence information that has beendetermined.

In the present embodiment, as described below, the decision of an objectin the object zone can be realized with high reliability compared to thecase where the synthesis is not performed, by establishing thecorrespondence between an object and an object zone through synthesizingthe characteristic quantity associated with the object zone fromcharacteristic quantities of objects. In particular, when establishingthe correspondence between an object and an object zone at the time whenthe state of crossover ceases, the characteristic quantities of aplurality of objects are synthesized, and the degree of similaritybetween the characteristic quantities of both the object zone and aplurality of objects, possibly still contained in an object zone afterpartition of the crossed objects, is calculated through the use of thesynthesized characteristic quantity, whereby it is prevented to cause anerroneous correspondence between objects and an object zone even when aplurality of objects are still contained in an object zone afterpartition of the crossed objects or when two groups of the objectscounterchange at the time of intersection (crossover).

Explanation is next presented regarding the operation of the presentembodiment with reference to the flow chart shown in FIG. 4. FIG. 4 is aflow chart illustrating an example of the process implemented by theobject-tracking device of the present invention.

Object-zone extraction section 12 receives image information throughimage input terminal 11 (Step S301). Object-zone extraction section 12extracts an object zone from the received image information (Step S302).Object-zone extraction section 12 next provides object zone informationincluding the image information about the object zone (object-zone imageinformation).

State-of-tracking decision section 13 stores the object zone informationprovided by object-zone extraction section 12. State-of-trackingdecision section 13 establishes the correspondence between an object andan object zone by tracking the object on the basis of the current objectzone information supplied from object-zone extraction section 12, thepast object zone information stored in state-of-tracking decisionsection 13 itself and the past correspondence information that has beendetermined supplied from correspondence decision section 16; and decidesthe state of tracking of the object and the object zone with each objector with each object zone. It is supplementarily remarked that the stateof tracking is represented as a state of tracking of an object whenattention is focused on an object, and is represented as a state oftracking of an object zone when attention is focused on an object zone.When, however, the correspondence between the object and object zone hasalready been established, both of the representations are equivalent.

In this way, state-of-tracking decision section 13 provides first zonecorrespondence information as an output (Step S303). The first zonecorrespondence information is the information that represents thecorresponding relationship between object and object zone and the stateof tracking (the information representing the correspondence of anobject zone and an object to the state of tracking). The correspondenceinformation that has been determined is the information representing thedecided corresponding relationship between an object and an object zonedecided by correspondence decision section 16. The procedure will bedescribed later, by which correspondence decision section 16 decides acorrespondence between the object and an object zone.

FIG. 5 is an explanatory drawing showing an example of the first zonecorrespondence information. The first zone-correspondence information isthe information that represents the correspondence of objects and theirstates of tracking to object zones as illustrated in FIG. 5A, which isintended to describe a correspondence between object zones α, β and γand objects A, B, C and D as illustrated in FIG. 5C where it is supposedthat the object zone information includes image information about threeobject zones α, β, γ and also four objects A, B, C and D. The firstzone-correspondence information can be the information that representsthe correspondence of object zones and their states of tracking torespective objects as illustrated in FIG. 5B.

The tracking of an object is to establish the correspondence of anobject to be tracked to an object zone. The establishment of thecorrespondence can be realized, for example, by setting up the positionof an object from an object zone corresponding to a past object, seekingthe object zone nearest the set-up position of the object andassociating the sought object zone with the object to be tracked. Inthis case, the position of an object can be assumed to be the center ofgravity of an object zone, for example. The method of establishing thecorrespondence will be described later.

Characteristic-quantity generation section 14 generates a characteristicquantity of an object zone on the basis of the image informationreceived through image input terminal 11 and the object zone informationreceived from object-zone extraction section 12 (Step S304) to supplyzone characteristic-quantity information to both characteristic-quantitysynthesis section 15 and correspondence establishment section 17. Zonecharacteristic-quantity information is the information that representsthe characteristic quantity of each object zone (a zone characteristicquantity) extracted by object-zone extraction section 12.Characteristic-quantity generation section 14 next decides thecharacteristic quantity of the object on the basis of the firstzone-correspondence information provided from state-of-tracking decisionsection 13 and the characteristic quantity of the object zone generatedin Step S304 to update the characteristic quantity of the stored object(Step S305). Characteristic-quantity generation section 14 next suppliesthe characteristic quantity of an object in response to the request ofcharacteristic-quantity synthesis section 15.

Characteristic-quantity generation section 14 further updates thecharacteristic quantity of an object when the first zone-correspondenceinformation provided from state-of-tracking decision section 13indicates that the state of tracking of the object is a stand-alonestate. Characteristic-quantity generation section 14 does not effect theupdate of the stored characteristic quantity when the state of trackingis a non-stand-alone state (the state other than the stand-alone state),because an exact calculation of the characteristic quantity of an objectis unable when the object intersects with another object in the image.The characteristic quantity is updated according to formula (1), forexample.H _(t)=(1−η)H _(t−1) +?ηH _(in) . . . (0≦η≦1)  (1)where H_(in) represents a currently observed characteristic quantity ofan object (the characteristic quantity obtained from a set of objectzones currently constituting an object). H_(t) and H_(t−1) represent thecharacteristic quantities of an object stored at times t and t−1,respectively, of those quantities stored in characteristic-quantitygeneration section 14, H_(t) representing a characteristic quantity ofan object after the update. The symbol η denotes the coefficient ofupdate for the characteristic quantity of an object.

Characteristic-quantity synthesis section 15 synthesizes, bycalculation, a characteristic quantity from the characteristicquantities of objects for each of all necessary combinations of aplurality of objects (hereinafter, referred to as a synthesizedcharacteristic quantity), on the basis of both the object characteristicquantities stored in characteristic-quantity generation section 14 andthe first zone-correspondence information provided fromstate-of-tracking decision section 13 (Step S306). In other words,characteristic-quantity synthesis section 15 decides all possiblecombinations of objects and object zones based on object characteristicquantities provided by characteristic-quantity generation section 14 andthe first zone-correspondence information, and generates synthesizedcharacteristic quantities (it is postulated that the synthesizedcharacteristic quantity is a concept that involves an objectcharacteristic quantity of a single object) only for the decidedcombinations of the objects and object zones. Characteristic-quantitysynthesis section 15 supplies the synthesized characteristic-quantityinformation inclusive of the calculated synthesized characteristicquantities. The synthesized characteristic-quantity information refersto the information that includes the calculated synthesizedcharacteristic quantities and the correspondences between the objectsand the synthesized characteristic quantities employed for calculatingthe synthesized characteristic quantities. Furthermore,characteristic-quantity generation section 14 selects storedcharacteristic quantities of the objects in response to the requests ofcharacteristic-quantity synthesis section 15 to provide the selectedobject characteristic quantities to characteristic-quantity synthesissection 15. Characteristic-quantity synthesis section 15 determines theobject characteristic quantities to request to characteristic-quantitygeneration section 14 on the basis of the first zone-correspondenceinformation.

All necessary combinations of a plurality of objects are determined asfollows: suppose, for example, the case where objects A, B and C arebrought into intersection (i.e., the state of crossover) and then becomeparted into two object zones α and β (i.e., the state of parting); then,eight possible combinations of the objects and object zones areenvisaged, i.e. (α, β)=(ABC, Φ), (Φ, ABC), (A, BC), (B, CA), (C, AB),(BC, A), (CA, B), and (AB, C). Characteristic-quantity synthesis section15 generates synthesized characteristic quantities for A, B, C, AB, BC,CA, and ABC. In the above description, (α, β)=(AB, C), for example,represents objects A and B residing in object zone α and object Cresiding in object zone β. The symbol Φ represents no object in thecorresponding object zone. The synthesized characteristic-quantityinformation includes, for example, synthesized characteristic quantityof AB and the information representing the fact that the objectscorresponding to the calculated synthesized characteristic-quantity ofAB are object A and object B.

Correspondence decision section 16 decides the correspondence betweenobjects and object zones on the basis of zone characteristic-quantityinformation provided by characteristic-quantity generation section 14,synthesized characteristic-quantity information provided bycharacteristic-quantity synthesis section 15 and the firstzone-correspondence information provided by state-of-tracking decisionsection 13 (Step S307). Correspondence decision section 16 next suppliesthe correspondence information that has been determined.

Correspondence decision section 16 decides the correspondence relationbetween objects and object zone for the objects when the state oftracking of the objects exhibits a state of parting in the firstzone-correspondence information supplied from state-of-tracking decisionsection 13, on the basis of the zone characteristic-quantity informationand the synthesized characteristic-quantity information. Alternatively,for the objects of a state of tracking, indicated as anon-state-of-parting state (the state other than the state of parting)in the first zone-correspondence information, correspondence decisionsection 16 decides the correspondence between objects and object zone toaccord with the information representing the correspondence to theobject zones included in the first zone-correspondence informationsupplied from state-of-tracking decision section 13. Correspondencedecision section 16 employs the information about the decidedcorrespondence between the objects and object zones as thecorrespondence information that has been determined.

State-of-tracking decision section 13 decides which of a stand-alonestate, a crossover state and a state of parting the states of trackingof the objects exhibit by means of the method described bellow.State-of-tracking decision section 13 classifies overall objects andobject zones on the basis of current object-zone information providedfrom object-zone extraction section 12, the past object-zone informationstored in state-of-tracking decision section 13 itself, and the pastdecided-correspondence information provided by correspondence decisionsection 16.

If correspondence information that has been determined indicates anestablished correspondence between object A and object zone α, forexample, then state-of-tracking decision section 13 enrolls object A andobject zone α in class X. If object B and object zone β are associatedinto an established correspondence relation, then state-of-trackingdecision section 13 enrolls object B and object zone β in class Y otherthan class X. Furthermore, if object A and object B have an establishedcorrespondence to γ as well, then state-of-tracking decision section 13,from the view that there is an established correspondence between objectA belonging to class X and object B belonging to class Y, enrolls objectzone γ, all elements belonging to class X and all elements belonging toclass Y in a same class. For example, classification can be made toenroll object zone γ and both object B and object zone β, which are theelements belonging to class Y, to in class X. Consequently, class Y isdeleted, and objects A and B, object zones α, β, and γ belong to classX. The classification can thus be implemented such that within oneclass, each object and each object zone associate with at least oneother object or other object zone in corresponding relationship.

State-of-tracking decision section 13, subsequently to theclassification of the objects and object zones, decides which of thestand-alone state, crossover state and state of parting the state oftracking is now taking. State-of-tracking decision section 13 decidesthat: if a single object element is in the elements of a class, then thestate of tracking of the object is the stand-alone state; if two or moreobject elements are in the elements of a class, and also if only oneobject-zone element is present in the elements of the class, then theoverall state of tracking of the two or more object elements is thecrossover state; and if two or more object elements are present in theelements of a class and further two or more object-zone elements arepresent in the elements of the class, then the overall state of trackingis the state of parting.

Explanation next regards a sequence of procedures from the synthesis ofcharacteristic quantities to the selection of object zones and objects.Correspondence decision section 16 calculates the degree of combinedsimilarity for each of all combinations of objects and object zones andselects the combination of the objects and object zone that has thehighest degree of combined similarity. The selection of combinationleads to the establishment of correspondence between an object and anobject zone. In the above description, the degree of combined similarityis an index indicative of the validness of the combination obtained fromthe degrees of similarity between the objects and object zones that makeup the combination. A high degree of combined similarity represents avalid combination as well as the similarity in the characteristics ofthe object and object zone that make up the combination. It is to benoted that while a totaled degree of similarity, i.e. the sum of thedegrees of similarity between characteristic quantities of object zonesand synthesized characteristic quantities for respective combinations,is used as the degree of combined similarity, the degree of combinedsimilarity is not limited to the sum of the degrees of similarity.

For example, in the case when objects A, B and C are brought intointersection (i.e., the state of crossover) and then become parted, thedegrees of combined similarity can be calculated from characteristicquantities of object zones and synthesized characteristic quantities forthe eight combinations of objects and object zones, (α, β)=(ABC, Φ), (Φ,ABC), (A, BC), (B, CA), (C, AB), (BC, A), (CA, B), and (AB, C).Correspondence decision section 16 decides that the combination of theobject and object zone having the highest calculated degree of combinedsimilarity is the object and object zone of the optimum correspondence.

For example, suppose the case where it is determinately impossible toassociate object C and object zone α in corresponding relationship aswith the case where the objects that potentially correspond to objectzone α are A and B while the objects that potentially correspond toobject zone β are A, B and C. In this case, correspondence decisionsection 16 may calculate the degrees of combined similarity for onlyfour combinations, i.e. (α, β)=(AB, C), (Φ, ABC), (A, BC), (B, CA),whereby correspondence decision section 16 can decrease the amount ofcalculation for the similarity calculation.

In order to calculate the degrees of combined similarity, correspondencedecision section 16 calculates the distance between the synthesizedcharacteristic quantity of objects and the characteristic quantity ofthe object zone (hereinafter referred to as a distance betweencharacteristic quantities or an inter-characteristic-quantity distance)and then calculates a combination distance from the distances betweencharacteristic quantities. Correspondence decision section 16 decidesthat the degree of combined similarity is high if the calculatedcombination distance is small, and that the degree of combinedsimilarity is low if the calculated combination distance is large. Thecombination distance is calculated to be, for example, a sum of thedistances between characteristic quantities as shown in expression (2).$\begin{matrix}{D_{k} = {\sum\limits_{i}\quad{{{Hr}_{i} - {Hg}_{ki}}}^{2}}} & (2)\end{matrix}$where i denotes the number of object zone, k the number of thecombination of object and object zone, Hr_(i) the characteristicquantity of the i-th object zone, and Hg_(ki) the synthesizedcharacteristic quantity of the objects corresponding to the i-th objectzone in the k-th combination. D_(k) stands for the combination distancefor the k-th combination.

In the above-described eight combinations of objects and object zones,(α, β)=(ABC, Φ), (Φ, ABC), (A, BC), (B, CA), (C, AB), (BC, A), (CA, B),and (AB, C), i=1 and 2 and k=1 through 8. Correspondence decisionsection 16 calculates the combination distance (D_(k)) for each of allpossible combinations (in this example, eight combinations) from thedistances between the characteristic quantities in each of the sets,each being made up of objects (or an object) and an object zone, (forexample, in the combination (α, β)=(A, BC), one set being made up of aand A and another set being made up of β and BC, and the combinationdistance (D_(k)) being the sum of the distance between a and A and thedistance between β and BC); and determines the combination of theminimum combination distance from the possible combinations, as acombination of the highest similarity.

Characteristic-quantity synthesis section 15 preferably calculates thesynthesized characteristic quantity depending on a desired synthesisratio ε. Correspondence decision section 16 calculates the degree ofcombined similarity on the basis of the synthesized characteristicquantity calculated depending on the synthesis ratio ε to select theoptimum combination of objects (or an object) and an object zone. Thesynthesis ratio ε refers to a coefficient for adjusting the ratio of thecharacteristic quantity of each object contained in an object zone. Forexample, apparent magnitudes of objects displayed in the image differ insome situations on account of different distances of the photographicsubjects from a camera. In such a case, the calculation of thesynthesized characteristic quantity through simply summing thecharacteristic quantities of objects will lead correspondence decisionsection 16 to decision of an incorrect correspondence between objects oran object and an object zone, because the calculated synthesizedcharacteristic quantity differs from the actual characteristic quantityof the object zone. Correspondence decision section 16 can prevent anincorrect correspondence between objects or an object and an object zoneby allowing characteristic-quantity synthesis section 15 to calculate asynthesized characteristic quantity depending on a predeterminedsynthesis coefficient ε.

When synthesizing characteristic quantities of two objects A and B, forexample, characteristic-quantity synthesis section 15 generates asynthesized characteristic quantity according to the formula (3), andcorrespondence decision section 16 calculates the combination distanceaccording to the formula (4).Hg(e)=(1−ε)H _(A) +εH _(B)(0≦ε≦1)  (3) $\begin{matrix}{D_{k} = {\sum\limits_{i}\quad{\min\limits_{ɛ}{{{Hr}_{i} - {{Hg}_{ki}(ɛ)}}}^{2}}}} & (4)\end{matrix}$

Characteristic-quantity synthesis section 15 varies synthesiscoefficient ε from 0 through 1 (for example, from 0 through 1 atintervals of 0.1) and calculates synthesized characteristic quantitieson the basis of all values of the varied synthesis coefficient ε.Correspondence decision section 16 selects the synthesizedcharacteristic quantity that minimizes the inter-characteristic-quantitydistance between the characteristic quantity of an object zone and thesynthesized characteristic quantity of objects with respect to thevariation of ε. Correspondence decision section 16 further calculatesthe combination distance for each combination (k) in accordance withformula (4) on the basis of the selected synthesized characteristicquantity and defines the combination of the objects and the object zonethat makes the combination distance minimum as an optimumcorrespondence.

Although correspondence decision section 16 calculates theinter-characteristic-quantity distance in terms of an L2 norm, any othermeasure of distance may be used for the calculation of theinter-characteristic quantity if the calculation technique supports thecalculation of the degree of similarity. An L2 norm refers to a measureof distance calculated in the form of the sum of squares of differencesas shown in formulas (2) and (4).

As described above, the device for tracking an object of the presentinvention enables continuation of tracking a target object even when,after a plurality of objects intersect and then part, a set of pluralobjects still exists, or when the combinations of the intersectingobjects counterchange immediately before and after the intersection ofthe objects takes place, by calculating synthesized characteristicquantities of the objects in crossover states; calculating the degreesof similarity between the characteristic quantities of the object zonesand the synthesized characteristic quantities; and deciding thecorresponding relationship between the objects and object zones based onthe calculated degrees of similarities.

Furthermore, the device for tracking an object of the present inventioncalculates the degrees of similarity between the characteristicquantities of object zones and the synthesized characteristic quantitiesfor all combinations of objects and object zones and selects an optimumcombination of the objects and object zones based on the calculateddegrees of similarity, thereby allowing selection of an optimumcorrespondence of overall objects when establishing the correspondencerelation between a target object and the zone where the target objectexists.

Detailed explanation is next presented regarding the constructions ofstate-of-tracking decision section 13, characteristic-quantitygeneration section 14 and correspondence decision section 16. Theexplanation first regards the construction of state-of-tracking decisionsection 13. FIG. 6 is a block diagram illustrating an example of theconstruction of state-of-tracking decision section 13. In the exampleshown in FIG. 6, state-of-tracking decision section 13 comprises objecttracking section 131, state decision section 132 andobject-zone-information storage section 133.

Object tracking section 131 is an embodiment of an object tracking meansfor tracking an object or objects through the use of the object zoneinformation, correspondence information that has been determined and thepast object zone information provided from an object-zone-informationstoring means, to provide a second zone-correspondence information thatindicates the corresponding relationship between objects and objectzones. Object-zone-information storage section 133 is an embodiment ofan object-zone-information storing means for storing the object-zoneinformation and the first zone-correspondence information. Statedecision section 132 is an embodiment of a state decision means fordeciding the state of tracking of an object from the secondzone-correspondence information, object-zone information and pastobject-zone information, to provide the first zone-correspondenceinformation.

Object tracking section 131 tracks objects and decides correspondencesbetween the objects and the object zones on the basis of currentobject-zone information provided by object-zone extraction section 12,past object-zone information stored in object-zone-information storagesection 133 and past correspondence information that has been determinedprovided by correspondence decision section 16. Object tracking section131 further generates the second zone-correspondence information anddelivers the generated second zone-correspondence information to statedecision section 132.

The second zone-correspondence information refers to the informationindicative of the corresponding relationship between objects and objectzones. Unlike the first zone-correspondence information, the secondzone-correspondence information includes no information about a state oftracking. FIG. 7 is an explanatory diagram illustrating an example ofthe second zone-correspondence information. For example, the secondzone-correspondence information is the information that indicates thecorrespondences between objects and object zones as is shown in FIG. 7Ato indicate the correspondences between object zones α β γ and objectsA, B, C and D as shown in FIG. 7C when it is recognized that theobject-zone information has image information about three object zonesα, β, γ and the past correspondence information that has been determinedhas four objects A, B, C and D: Alternatively, the secondzone-correspondence information can be the information that indicatesthe correspondences between the objects and object zones as shown inFIG. 7B, for example.

State decision section 132 performs the decision of the state oftracking objects on the basis of the object-zone information provided byobject-zone extraction section 12 and the second zone-correspondenceinformation provided by object tracking section 131, and supplies thefirst zone-correspondence information about the basis of the secondzone-correspondence information and the decision result.

Object-zone-information storage section 133 stores the object-zoneinformation provided by object-zone extraction section 12. Theobject-zone information stored in object-zone-information storagesection 133 serves as past object-zone information used for generationof the second zone-correspondence information in the next generationcycle.

FIG. 8 is a block diagram illustrating an example of the construction ofcharacteristic-quantity generation section 14. In the example shown inFIG. 8, characteristic-quantity generation section 14 comprisescharacteristic-quantity extraction section 141, characteristic-quantityupdate section 142 and characteristic-quantity storage section 143.

Characteristic-quantity extraction section 141 is an embodiment of acharacteristic-quantity extracting means for extracting a zonecharacteristic quantity from image information, object-zone informationand the first zone-correspondence information to provide the zonecharacteristic-quantity information that is the information includingthe zone characteristic quantities. Characteristic-quantity storagesection 143 is an embodiment of a characteristic-quantity storing meansfor selecting the stored object characteristic quantity as required andsupplying the selected object characteristic quantity.Characteristic-quantity update section 142 is an embodiment of acharacteristic-quantity updating means for updating the objectcharacteristic quantity stored in the characteristic-quantity storingmeans on the basis of the zone characteristic quantity, the firstzone-correspondence information and an object characteristic quantitygenerated prior to the present.

Characteristic-quantity extraction section 141 calculates acharacteristic quantity of each object zone on the basis of the imageinformation received by way of image input terminal 11 and theobject-zone information provided by object-zone extraction section 12,and provides, as an output, the zone characteristic-quantity informationthat includes the calculated characteristic quantities of object zones.The zone characteristic-quantity information provided as an output isreceived by both characteristic-quantity update section 142 andcorrespondence decision section 16.

For an object that is indicated in the first zone-correspondenceinformation as being in a stand-alone state in its state of tracking,characteristic-quantity update section 142 updates the characteristicquantity of the object stored in characteristic-quantity storage section143 based on the zone characteristic-quantity information provided fromcharacteristic-quantity extraction section 141. Characteristic-quantityupdate section 142 implements no update of the characteristic quantitystored in characteristic-quantity storage section 143 for the objectthat is indicated in the first zone-correspondence information as beingin the state other than the stand-alone state in its the state oftracking. The characteristic quantities of the objects stored bycharacteristic-quantity storage section 143 are used bycharacteristic-quantity synthesis section 15 for generating thesynthesized characteristic quantities: the stored characteristicquantity of an object is selected in response to the request ofcharacteristic-quantity synthesis section 15 and the selectedcharacteristic quantity of the object is provided tocharacteristic-quantity synthesis section 15.

FIG. 9 is a block diagram illustrating an example of the construction ofcorrespondence decision section 16. In the example shown in FIG. 9,correspondence decision section 16 comprises correspondence calculationsection 161 and correspondence establishment section 162.

Correspondence calculation section 161 is an embodiment of acorrespondence calculation means for generating the optimumcorrespondence information that represents an optimum correspondencebetween an object and an object zone. Correspondence establishmentsection 162 is an embodiment of a correspondence deciding means forproviding the correspondence information that has been determinedwherein the correspondence information that has been determined is theinformation including the decided correspondence relation between anobject and an object zone.

Correspondence calculation section 161 calculates the degrees ofsimilarities between the characteristic quantities of object zones andsynthesized characteristic quantities and further calculates the degreesof combined similarity for all combinations possibly to be placed incorresponding relationship, of the combinations of objects and objectzones, on the basis of the zone characteristic-quantity informationprovided by characteristic-quantity generation section 14, thesynthesized characteristic-quantity information provided bycharacteristic-quantity synthesis section 15, and the firstzone-correspondence information provided by state-of-tracking decisionsection 13. Correspondence calculation section 161 further implements aselection to select as the optimum correspondence the combination ofobjects and object zones that has the highest degree of combinedsimilarity to generate the optimum correspondence information andprovides the optimum correspondence information as an output. In theabove description, the optimum correspondence information refers to theinformation that indicates the optimum corresponding relationshipbetween objects and object zones (the combination of objects and objectzones having the highest degree of combined similarity).

Correspondence establishment section 162 determines to designate theinformation about the correspondence between objects and object zonesincluded in the the optimum correspondence information provided bycorrespondence calculation section 161, as the information aboutcorrespondences between the objects and object zones, if the firstzone-correspondence information provided by state-of-tracking decisionsection 13 indicates that the state of tracking of the objects is thestate of parting. For the objects indicated as the objects of the stateother than the state of parting, correspondence establishment section162 establishes the information about the correspondences between theobjects of interest and object zones included in the firstzone-correspondence information provided by state-of-tracking decisionsection 13 as the information about correspondences between the objectsof interest and object zones.

Correspondence establishment section 162, furthermore, compares theinformation about objects included in the first zone-correspondenceinformation and the information about established correspondence betweenobjects and object zones, and when there comes out an object zone thatis not in correspondence to any of the objects included in the firstzone-correspondence information (will be referred to as anot-in-correspondence object zone), creates a new object, establishes acorrespondence of the new object to the not-in-correspondence objectzone, and supplies the information about the established correspondencebetween the object and object zone as the correspondence informationthat has been determined.

Suppose, for example, the case where two objects A and B are present ina completely superposed configuration within a single object zone α sothat an object-tracking device recognizes only one object A in objectzone α. In this situation, when the state of objects A and B makestransition to the state of parting, a new object zone β parts fromobject zone α as a not-in-correspondence object zone. Theobject-tracking device, however, has not recognized the existence ofobject B, and, as a result, there comes out an object zone β that has nocorrespondence to any object. For this reason, correspondenceestablishment section 162 creates new object B and associates object Bwith object zone β in corresponding relationship.

It is to be noted that the constructions illustrated in FIGS. 2, 3, 6, 8and 9 are feasible by means of software. Specifically, theobject-tracking device of the present embodiment can be realized bymeans of an object-tracking program for a computer to implement thefollowing procedures: image input processing to receive imageinformation; object-zone extraction processing for extracting objectzones from the input image information to provide object-zoneinformation inclusive of the image information about the object zones;state-of-tracking decision processing for deciding the states oftracking for each of the objects or for each of the object zones fromthe object-zone information and the correspondence information that hasbeen determined indicative of the corresponding relationship between theobject zones and objects prior to the present and providing a firstzone-correspondence information indicative of the correspondingrelationship between the object zones and objects and the states oftracking; characteristic-quantity generation processing for generatingthe zone characteristic quantities representative of characteristicquantities of object zones and the object characteristic quantitiesrepresentative of characteristic quantities of objects based on theimage information, the object-zone information and the result of thedecision provided by the state-of-tracking deciding means;characteristic-quantity synthesis processing for generating each ofsynthesized characteristic quantities by synthesizing characteristicquantities for all the required combinations of a plurality of objectsbased on the object characteristic quantities and the firstzone-correspondence information and providing the synthesizedcharacteristic-quantity information that is the information inclusive ofthe synthesized characteristic quantity and the correspondingrelationship between objects and the synthesized characteristicquantities employed for generating the synthesized characteristicquantities; and correspondence establishment processing for establishingthe correspondences between objects and object zones based on theobject-zone information, the first zone-correspondence information, thezone characteristic-quantity information and the synthesizedcharacteristic-quantity information and supplying the currentcorrespondence information that has been determined.

Modified Embodiment 1

State-of-tracking decision section 13 may decide the state of trackingof an object in accordance with the method of decision described below.For example, in the case where, due to superposition of objects (thecrossover state), only one object is at first put in correspondence toan object zone, it is decided that the state of tracking is astand-alone state. When the object zone is parting into a plurality ofobject zones from the above described state, there exists only oneobject. As a result, it may not be decided that the state of trackinghas made transition from the crossover state to the state of parting butmay be possibly decided that the state of tracking remains in astand-alone state. In order to cope with the situation that one objectin a stand-alone state is parting into a plurality of objects,state-of-tracking decision section 13 can decide that the state oftracking should be the state of parting, provided that two or moreobject-zone elements are included in a class even if there is only asingle object element in the class.

The above-described embodiment shows an example of the method ofextracting object zones in which object-zone extraction section 12calculates the differences in pixel values between the image informationsupplied by way of image input terminal 11 and the background imageinformation that has been acquired in advance. In some cases, whencalculating the differences of pixel values, object-zone extractionsection 12 cannot properly extract the object zone on account of theobject having characteristics similar to those of the background image(for example, the object and the background image resembles each otherin color). If the object zone is not properly extracted, state decisionsection 132 of state-of-tracking decision section 13 might decide bymistake that the state of tracking has made transition to the state ofparting. In order to achieve a proper extraction of the object zone insuch a case, state decision section 132 preferably copes with theproblem through the use of a spatial distance between object zones.

Specifically, for the object zones that are situated as near as within apredetermined threshold of an inter-object-zone distance, state decisionsection 132 sorts, into the same class, object zones and the objectsthat are associated with each of the object zones in correspondencerelation, when classifying objects and object zones. Suppose, forexample, the case where there are objects A and B and object zones α andβ with object zone α corresponding to object A and object zone βcorresponding to object B. According to the method explained in theabove-described embodiment, state decision section 132 defines twoclasses: the class to which belong object zone α and object A and theclass to which belong object zone β and object B. In contrast to theabove-described embodiment, state decision section 132 can group all ofobject zones α, β and objects A, B together in a single class, providedthat the distance between object zone α and object zone β is no largerthan a predetermined threshold.

The grouping into a single class allows state decision section 132 todecide that the state of tracking remains unchanged in the crossoverstate even when an object zone parts into two or more object zones inthe class that includes two or more objects, provided that theinter-object-zone distance is no larger than a predetermined threshold.State decision section 132 decides that the state of tracking hasentered the state of parting at the time when the inter-object-zonedistance surpasses the predetermined threshold.

Modified Embodiment 2

Furthermore, it is intended that state decision section 132 can copewith the case where a part of an object zone accidentally parts,depending on a period of on-parting duration of the object zones. Theduration period for on-parting refers to a duration period during whichobject zones are in a parting-enabled state, wherein the parting-enabledstate of object zones is defined as a state that, when an object zonethat was formerly a single object zone parts into the object zones, theparted object zones keep the inter-object-zone distance or distances inexcess of a prescribed threshold (can be 0). In this case, statedecision section 132 groups the object zones in the process of continuedperiods of separation shorter than a predetermined threshold and alsothe objects corresponding to the object zones into the same class toreserve the former state of tracking. When the duration period foron-parting exceeds the threshold, state decision section 132 decidesthat a state of tracking is the state of parting.

The duration period for on-parting is updated and stored as follows: theduration period for on-parting is stored in object-zone-informationstorage section 133; object tracking section 131 acquires the continuedperiods of separation stored in object-zone-information storage section133, correlates the duration period for on-parting with each of objectzones and calculates a new duration period for on-parting; and objecttracking section 131 updates the continued periods of separation storedin object-zone-information storage section 133.

Object tracking section 131 makes a correlation between the durationperiod for on-parting and an object zone as follows: object trackingsection 131 makes a correlation between a current object zone and a pastobject zone on the basis of a current object zone included in theobject-zone information, provided from object-zone extraction section12, and the past correspondence information that has been determined,provided from correspondence decision section 16; and object trackingsection 131 correlates the duration period for on-parting, correlatedwith the past object zone stored in object-zone-information storagesection 133, with the current object zone that corresponds to the pastobject zone of interest. If a plurality of continued periods ofseparation are correlated, object tracking section 131 correlates thelongest of the plurality of continued periods of separation with theobject zone. In addition, object tracking section 131 can correlate theshortest of the plurality of continued periods of separation with acurrent objection zone. Alternatively, object tracking section 131calculates an average of a plurality of continued periods of separationand correlates the calculated average period with the current objectzone.

State decision section 132 acquires continued periods of separation fromobject-zone-information storage section 133. Furthermore, state decisionsection 132 decides the state of tracking of an object zone. Further,based on the decision result, state decision section 132 updatescontinued periods of separation stored in object-zone-informationstorage section 133. In more specified terms, state decision section132, when having decided that the state of tracking is a parting-enabledstate, updates continued periods of separation stored inobject-zone-information storage section 133 on the basis of the durationperiod for on-parting of interest and the current time. State decisionsection 132 resets the duration period for on-parting if the state oftracking of the object zone is not the parting-enabled state.Alternatively, parting-startup time representative of the startup timeof parting may also be stored in place of storing the duration periodfor on-parting. In this case, the duration period for on-parting can beobtained by finding the difference between the current time andparting-startup time.

Object tracking section 131 can generate the second zone-correspondenceinformation including the duration period for on-parting rather thanstoring the duration period for on-parting in object-zone-informationstorage section 133. In this case, state decision section 132 acquiresthe duration period for on-parting from the second zone-correspondenceinformation. The direct communication of the duration period foron-parting from object tracking section 131 to state decision section132 without interposing object-zone-information storage section 133offers an advantage of allowing reduction in the amount of processingthrough object-zone-information storage section 133.

Still furthermore, the first embodiment and the modified embodiments 1and 2 may be employed in combination. In concrete terms, it is alsopracticable to bring together the objects to which the object zoneshaving a common region correspond, on the basis of the combination ofthe correspondences between objects and object zones, the distancesbetween object zones and the duration periods of parting of object zonesand to group the objects and corresponding object zones into one class.Further, it is also practical to decide that the states of tracking ofobjects and object zones included in a class are the states of parting,provided that any one of the following conditions or the combination ofthe following conditions is met: the classified classes meet thecondition for the state of parting and further two or more objects areincluded in each of the classes; the distance between the object zonesincluded in each class exceeds a predetermined threshold; the durationtime during the object zones being in the state of parting exceeds apredetermined threshold.

Modified Embodiment 3

The object-tracking device can synthesize the characteristic quantitiesof objects to establish the correspondences between objects and objectzones through the following procedures: characteristic-quantitysynthesis section 15 acquires not only the characteristic quantities ofobjects but also the zone characteristic-quantity information fromcharacteristic-quantity generation section 14. Characteristic-quantitysynthesis section 15 calculates synthesized characteristic quantitiesdepending on a desired synthesis ratio ε on the basis of the zonecharacteristic-quantity information and characteristic quantities of theobjects provided by characteristic-quantity generation section 14.Characteristic-quantity synthesis section 15 further provides thesynthesized characteristic quantities corresponding to the synthesisratio ε that yields the highest degree of similarity between thecalculated synthesized characteristic quantity and the characteristicquantity of the object zone included in the zone characteristic-quantityinformation. Correspondence decision section 16 calculates the degree ofsimilarity between the characteristic quantities of object zones and thesynthesized characteristic quantities for each combination of objectsand object zones based on the synthesized characteristic quantitiesprovided by characteristic-quantity synthesis section 15 and calculatesthe degrees of combined similarity.

Since characteristic-quantity synthesis section 15 provides thesynthesized characteristic quantity corresponding to the synthesis ratioε that yields the highest degree of similarity between objects and anobject zones, it suffices for correspondence decision section 16 tocalculate the synthesized characteristic quantity on the basis of onlyone synthesized characteristic quantity for each of the combinationsbetween the objects and object zones. Consequently, correspondencedecision section 16 can cut off redundant calculation processing.

Modified Embodiment 4

The object-tracking device can establish the correspondences betweenobjects and object zones by synthesizing characteristic quantitiesthrough the following method: characteristic-quantity synthesis section15 acquires not only the characteristic quantities of objects but alsothe zone characteristic-quantity information and calculates the value ofsynthesis ratio ε that minimizes the distance according to formulas (5)through (7), wherein formula (5) is a formula for calculating thedistance f(ε) between a characteristic quantity of an object zone andthe synthesized characteristic quantity of the corresponding objects onthe basis of the zone-correspondence information and the characteristicquantities of objects provided by characteristic-quantity generationsection 14; characteristic-quantity synthesis section 15 calculates thevalue of synthesis ratio E that nulls the function generated bypartial-differentiating distance f(ε) with respect to synthesis ratio εas shown in mathematical expressions (6) and (7), i.e.characteristic-quantity synthesis section 15 calculates the value ofsynthesis ratio that minimize distance f(ε); characteristic-quantitysynthesis section 15 generates a synthesized characteristic quantityaccording to formula (3) through the use of the calculated synthesisration ε; and correspondence decision section 16 calculates the degreeof similarity on the basis of the generated synthesized characteristicquantity. It is remarked that formulas (5) through (7) are expressedusing an example of the synthesized characteristic quantity for objectsA and B. $\begin{matrix}\begin{matrix}{{f(ɛ)} = {{{Hr}_{i} - {{Hg}_{ki}(ɛ)}}}^{2}} \\{= {{{\left( {H_{A} - H_{B}} \right)ɛ} + {Hr}_{i} - H_{A}}}^{2}} \\{= {\sum\limits_{n}\quad\left\{ {{\left( {a_{n} - b_{n}} \right)ɛ} + x_{n} - a_{n}} \right\}^{2}}}\end{matrix} & (5) \\{\frac{\partial f}{\partial ɛ}2{\sum\limits_{n}\quad{\left\{ {{\left( {a_{n} - b_{n}} \right)ɛ} + x_{n} - a_{n}} \right\}\left( {a_{n} - b_{n}} \right)}}} & (6) \\{ɛ = \frac{\sum\limits_{n}\quad\left( {a_{n} - x_{n}} \right)}{\sum\limits_{n}\quad\left( {a_{n} - b_{n}} \right)}} & (7)\end{matrix}$where Hg_(ki) denotes the synthesized characteristic quantity ofcharacteristic quantity H_(A) of object A and characteristic quantityH_(B) of object B, and Hr_(i) denotes the characteristic quantity of theobject zone. If the characteristic quantity is, for example, a colorhistogram and if the number of the ranks (BIN) of the color histogram isn, then characteristic quantities H_(A), H_(B), Hr_(i) and Hg_(ki) aren-dimensional vectors, respectively and are represented as H_(A)=(a₁,a₂, . . . , a_(n)), H_(B)=(b₁, b₂, . . . b_(n)), Hr_(i)=(X₁, x₂, . . .x_(n)), and Hg_(ki)=(x_(k1), x_(k2), . . . , x_(kn)).

If synthesis ratio ε calculated according to formula (7) is eithersmaller than 0 or larger than 1, then either 0 or 1 that is nearer thevalue of synthesis ratio ε calculated in accordance with formula (7) isselected as the value of synthesis ratio ε. Characteristic-quantitysynthesis section 15 calculates the synthesized characteristic quantitythrough the use of calculated synthesis ratio ε. Based on the calculatedsynthesis ratio ε, correspondence decision section 16 calculates thedistance of each combination represented by formula (2) and determinesthe combination of object and object zone which minimizes the distanceof combination as an optimum correspondence.

According to formulas (5), (6) and (7), characteristic-quantitysynthesis section 15 can generate a synthesized characteristic quantityusing exclusively a specific synthesis coefficient ε. Consequently, itis possible to reduce the amount of calculation for calculating thedegree of similarity, in comparison to the case in which synthesizedcharacteristic quantities are generated for various values of thesynthesis coefficients ε. It is to be noted that characteristic-quantitysynthesis section 15 can generate a synthesized characteristic quantityby the method analogous to the method described above as well in thecase when three or more objects superpose. In addition, whilecharacteristic-quantity synthesis section 15 employed the L2 norm as ameasure of a distance between characteristic quantities in theembodiment described above, another measure of a distance can beemployed.

Furthermore, the object-tracking device can be the device capable ofsynthesizing characteristic quantities and selecting object zones andobjects by the following method: characteristic-quantity synthesissection 15 generates synthesized characteristic quantities using ratiosof object areas as synthesis ratios ε; and correspondence decisionsection 16 calculates the degrees of similarity based on the generatedsynthesized characteristic quantities. This method is based on the factthat the syntheses ratio at the time of crossover of objects is nearlyequal to the ratio of the areas of the objects. Employing the area ratioof objects as a synthesis ratio enables characteristic-quantitysynthesis section 15 to cut down the operations for calculating thesynthesis ratio. The synthesized characteristic quantity is calculatedaccording to formula (8). $\begin{matrix}{{Hg} = {{\frac{S_{A}}{S_{A} + S_{B}}H_{A}} + {\frac{S_{B}}{S_{A} + S_{B}}H_{B}}}} & (8)\end{matrix}$where H_(A) and H_(B) denote the characteristic quantities of objects Aand B, respectively; S_(A) and S_(B) denote the areas on imageinformation about objects A and B, respectively; and Hg stands for thesynthesized characteristic quantity. Correspondence decision section 16calculates the sum of the distances represented in formula (2) throughthe use of the generated synthesized characteristic quantities Hg anddetermines the combination of objects and object zones that minimizesthe sum of the distances, as an optimum correspondence.

For example, characteristic-quantity generation section 14, taking thenumber of pixels included in an image of an object zone as an area ofthe object, generates a characteristic quantity inclusive of the area ofan object as well as the characteristic quantities inclusive of areas ofother objects. Characteristic-quantity storage section 143 incharacteristic-quantity generation section 14 stores the area-inclusivecharacteristic quantities and, when the state of tracking of an objectincluded in the first zone-correspondence information indicates thenon-crossover state (the state other than the crossover state), updatesthe area-inclusive characteristic quantity.

Furthermore, the object-tracking device can synthesize characteristicquantities and establish correspondences between objects and objectzones according to the method described below: presuming that the areaof each of superposed objects potentially changes in the range from S1through S2, and characteristic-quantity synthesis section 15 generatessynthesized characteristic quantities within the range of the changeaccording to formula (9). Formula (9) is intended to calculate thesynthesized characteristic quantity of the characteristic quantity ofobject A and the characteristic quantity of object B. Correspondencedecision section 16 calculates the degree of similarity between theobjects and the object zone based on the synthesized characteristicquantity included in the generated synthesized characteristic quantitiesand having the highest degree of similarity with respect to thecharacteristic quantity of the object zone, and calculates the degree ofcombined similarity, according to formula (10). $\begin{matrix}{{{Hg}({\Delta s})} = {{\frac{S_{A} + {\Delta\quad s}}{S_{A} + S_{B}}H_{A}} + {\frac{S_{B} - {\Delta\quad s}}{S_{A} + S_{B}}H_{B}\quad\left( {{- s_{1}} \leq {\Delta\quad s} \leq s_{2}} \right)}}} & (9) \\{D = {\sum\limits_{i}\quad{\min\limits_{\Delta\quad s}\left( {{Hr}_{i} - {{Hg}_{i}\left( {\Delta\quad s} \right)}} \right)^{2}}}} & (10)\end{matrix}$where ΔS denotes a difference of the area of object A from the area justbefore objects A and B superpose each other. Calculating of synthesizedcharacteristic quantities allowing for variations of the areas ofobjects enables characteristic-quantity synthesis section 15 to avoidthe selection of synthesis ratio quite different from practice even ifthe area of the object of interest more or less changes, therebyenabling the generation of the synthesized characteristic quantities onthe basis of correct synthesis ratios. As a result, correspondencedecision section 16 can decide an optimum correspondence between objectsand object zones.

Modified Embodiment 5

It is also feasible that characteristic-quantity synthesis section 15acquires not only the characteristic quantities of objects but also thezone characteristic-quantity information from characteristic-quantitygeneration section 14 and generates, in a range of the area variation,the synthesized characteristic quantities based on the zonecharacteristic-quantity information and the characteristic quantities ofobjects provided by characteristic-quantity generation section 14;provides as an output the synthesized characteristic quantity of thehighest degree of similarity to the characteristic quantity of an objectzone; and, based on the synthesized characteristic quantity,correspondence decision section 16 finds the degree of combinedsimilarity for each combination of objects and an object zones. In otherwords, the synthesis ratio ε can be limited to within a rangepredetermined on the basis of the range of the area variation of theobjects. In this way, characteristic-quantity synthesis section 15 needonly provide a single synthesized characteristic quantity for eachcombination of the objects, whereby redundant processing can be cut offin comparison to the case of generating synthesized characteristicquantities for various values of a synthesis ratio ε.

Modified Embodiment 6

Alternatively, characteristic-quantity synthesis section 15 can obtainthe synthesized characteristic quantities through the use of templatesof objects as characteristic quantities. The template refers to apartial image that is generated by extracting the part of the imagecorresponding to the material body of interest from a picture andcorresponds to the image that expresses the shape and the color. Atemplate is produced by extracting the zone in the picture thatcorresponds to a material body on the basis of the correspondencerelation between the object and the object zone. Alternatively, an imagethat has modelled in advance such as a reference face image can beemployed as a template.

For example, suppose the case where two objects reside in a singleobject zone. Then characteristic-quantity synthesis section 15 runstemplates across the object zone and calculates the degree of similaritybetween the template and the object zone at each position in the objectzone. Characteristic-quantity synthesis section 15 finds the positionwhere the degree of similarity attains a maximum value.Characteristic-quantity synthesis section 15 calculates the degree ofsimilarity according to formulas (11) and (12). $\begin{matrix}{{d_{i}\left( {a,b} \right)} = {\sum\limits_{{({{x + a},{y + b}})} \in m_{i}}\quad{{{M_{i}\left( {{x + a},{y + b}} \right)} - {R\left( {x,y} \right)}}}^{2}}} & (11) \\{d_{i} = {\min\limits_{a,b}\quad{d_{i}\left( {a,b} \right)}}} & (12)\end{matrix}$where M_(i)(x, y) represents the value of a pixel of the ith template atcoordinates (x, y) and R(x, y) represents the value of a pixel of theobject zone at coordinates (x y). Symbol m_(i) denotes the overalldomain of the coordinates of the ith template. Characteristic-quantitysynthesis section 15 calculates the sum of squares of the differences ofthe pixel values between the templates and the object zone, as shown informula (11), as the distance between their characteristic quantities.It is decided that the degree of similarity between the templates andthe object zone is the highest when the distance between theircharacteristic quantities is the minimum, as represented in formula(12). Characteristic-quantity synthesis section 15 takes the position(a, b) where the degree of similarity between the templates and theobject zone is the highest, as the position of the object.

Characteristic-quantity synthesis section 15 next decides the relationin the back-to-belly direction (the back/belly relation) of thesuperposed objects (the objects in the crossover state) based on thedegree of similarity between the template and the object zone.Characteristic-quantity synthesis section 15 decides that the templateof a higher degree of similarity to the object zone covers the templateof a lower degree of similarity to the object zone.Characteristic-quantity synthesis section 15 then generates asynthesized template that corresponds to a synthesized characteristicquantity by synthesizing the templates. As described above,characteristic-quantity synthesis section 15 can decide the back/bellyrelation of the respective objects on the basis of the characteristicquantities of the image templates and object zone and calculates asynthesized characteristic quantity by synthesizing the image templateson the basis of the decided back/belly relation of the objects.

Alternatively, characteristic-quantity synthesis section 15 can selectthe template that has the highest of the respective highest degrees ofsimilarity to the object zone of interest and decide that the selectedtemplate covers the unselected templates. It can be intended in thiscase that the region where the object zone and the selected template(the template in the case where the degree of similarity attains thehighest value) overlap is excluded from the object zone of interest and,on the basis of the remaining object zone and unselected templates,operations are repeatedly carried out for the calculation of the degreesof similarity between the templates and the object zone, the selectionof the template of the highest degree of similarity, the calculation ofthe position of the template that is of the highest degree ofsimilarity, and the decision of the covering template.

FIG. 10 is an explanatory diagram illustrating the concept of generatingsynthesized characteristic quantity using templates. As shown in FIG.10A, characteristic-quantity synthesis section 15 operates on templatesA and B to scan object zone α. Characteristic-quantity synthesis section15 can decide that template B covers template A in an overlappingconfiguration, because the highest degree of similarity between templateB and object zone α is higher than that between template A and objectzone α. As shown in FIG. 10B, characteristic-quantity synthesis section15 synthesizes template A and template B at the respective positionswhere each of the templates configures the highest degree of similarityto the object zone α, thereby generating synthesized template X.Characteristic-quantity synthesis section 15, when generatingsynthesized template X, performs the synthesizing operation using thevalue of the characteristic quantity of template B for the portion wheretemplate B overlaps template A. For supplements, the coordinates (a₁.b₁) stand for the position where the degree of similarity betweentemplate A and the object zone is the highest, the coordinates (a₂. b₂)standing for the position where the degree of similarity betweentemplate B and the object zone is the highest.

Correspondence decision section 16 calculates the degree of similaritybetween the characteristic quantity of the object zone and thesynthesized characteristic quantity based on the color histogram of thesynthesized template according to formula (2) and can calculate thedegree of combined similarity. Alternatively, correspondence decisionsection 16 can, analogously to formula (12), calculate the degree ofsimilarity between the characteristic quantity of the object zone andthe synthesized characteristic quantity through collating the objectzone and the synthesized template, thereby calculating the degree ofcombined similarity. Employing templates enables the correct calculationof the correspondence relation between objects and an object zone evenif an object is partially hidden behind another object.

Modified Embodiment 7

In each of above-described methods, when establishing the correspondencebetween objects and object zones, the case has sometimes occurred thatthe establishment of the correspondence between objects and the objectzones is necessarily processed by correspondence decision section 16even when the degree of the combined similarity is low, resulting in anincorrect correspondence between the objects and the object zones. Inview of this problem, correspondence decision section 16 can be intendednot to establish the correspondence between objects and object zones, ifthe degree of the combined similarity for an optimum combination (i.e.,the degree of the combined similarity for the combination of the highestdegree of combined similarity) is low. Correspondence calculationsection 161, in such a case, decides that, if the degree of the combinedsimilarity for the optimum combination of objects and object zones isequal to or smaller than a predetermined threshold, there are no optimumobject zones corresponding to the objects that constitute thecombination. Correspondence calculation section 161 then provides theoptimum correspondence information inclusive of the informationindicating the absence of the optimum correspondence between the objectsand the object zones. When the optimum correspondence informationincludes the information indicating the absence of the optimum objectzones corresponding to the objects, correspondence establishment section162 decides, for the objects of interest, to designate the informationindicating the correspondence between objects and object zones includedin the first zone-correspondence information as is, as thecorrespondence between the objects and the object zones and supplies theinformation as correspondence information that has been determined. Ifthis is not the case, the information indicating the correspondingrelationship between objects and object zones included in the optimumcorrespondence information is supplied as correspondence informationthat has been determined. Suppose, for example, the case wherecorrespondence calculation section 161 takes (α, β)=(A, BC) as anoptimum combination. Then correspondence calculation section 161calculates the degree of combined similarity from the degree ofsimilarity of the set made up of α and A and also the degree ofsimilarity of the set made up of β and BC and decides whether thedegrees of combined similarity are no larger than a predeterminedthreshold. If the degree of combined similarity is equal or smaller thanthe threshold, correspondence establishment section 162 decides todesignate the information about the corresponding relationship betweenobject A, object B and object C and object zones α and β included in thefirst zone-correspondence information as is, as the correspondencebetween object A, object B and object C and the object zones.

Modified Embodiment 8

In each of above-described methods, when establishing the correspondencebetween objects and object zones, the case has sometimes occurred thatthe establishment of the correspondences between objects and the objectzones is necessarily processed by correspondence decision section 16even when the degree of the combined similarity is low, entailing anincorrect establishment of the corresponding relationship between theobjects and the object zones. In view of this problem, correspondencedecision section 16 can be intended not to establish any correspondencesbetween objects and object zones, if the degree of the combinedsimilarity for an optimum combination (i.e., the degree of the combinedsimilarity for the combination of the highest degree of combinedsimilarity) is low. Correspondence calculation section 161, in such acase, decides that, if the degree of the set similarity (the degree ofsimilarity of a set made up of objects or an object and an object zone)for the optimum combination of objects and object zones is equal to orsmaller than a predetermined threshold, there are no optimum objectzones corresponding to the objects. Correspondence calculation section161 then provides the optimum correspondence information inclusive ofthe information indicating the absence of the optimum correspondencebetween the objects and the object zones. When the optimumcorrespondence information includes the information indicating theabsence of the optimum object zones corresponding to the objects,correspondence establishment section 162 decides to designate, for theobjects of interest, the information indicating the correspondingrelationship between objects and object zones included in the firstzone-correspondence information as is, as the corresponding relationshipbetween the objects and the object zones and supplies the information ascorrespondence information that has been determined. If it is not thecase, correspondence establishment section 162 supplies the informationindicating the corresponding relationship between objects and objectzones included in the optimum correspondence information ascorrespondence information that has been determined. Suppose, forexample, the case where correspondence calculation section 161 takes (α,β)=(A, BC) as an optimum combination. Then, correspondence calculationsection 161 decides whether or not the degree of similarity of the setmade up of α and A and also the degree of similarity of the set made upof β and BC are individually no higher than a predetermined threshold.If the degree of similarity between a and A is equal to or lower thanthe predetermined threshold, correspondence establishment section 162decides to designate the information indicating the correspondencebetween object A and the object zone of interest included in the firstzone-correspondence information as is, as the correspondence relationbetween object A and the object zone.

Modified Embodiment 9

It can be intended that correspondence calculation section 161establishes a correspondence to an object zone only for definitelydiscriminable objects. In this case, correspondence calculation section161 ranks all combinations of objects and object zones in the order ofheight in the degree of combined similarity. If the differences betweenthe degrees of combined similarity of the combinations ranked in theupper orders are small (for example, equal to or smaller than apredetermined threshold) and besides, if common objects and object zones(or an object and an object zone) are present in the upper-ordercombinations, correspondence calculation section 161 deems exclusivelythe combinations of the common objects and object zones as being of anoptimum correspondence. Correspondence calculation section 161 providesthe optimum correspondence information that includes the informationabout exclusively the combinations of the common objects and objectzones. This implies that the optimum correspondence information includesthe information that indicates the absence of any combination of anoptimum correspondence in the other objects and object zones.

Suppose, for example, the correspondences between objects A, B and C andobject zones α, β and γ in which the combinations ranked in the upperthree in the order of the degree of combined similarity are (α, β,γ)=(A, B, C), (AB, Φ, C), and (B, A, C). In this example, correspondencecalculation section 161 presumes exclusively the combination of object Cand object zone γ as an optimum combination, and provides the optimumcorrespondence information that includes the information indicative ofexclusively the combination of object C and object zone γ as an optimumcombination. In addition, this method can be combined with modifiedembodiment 7: In the case, where, for optimum combinations of theobjects and object zones, the degrees of similarity of the sets ofobjects and an object zone each are equal to or lower than apredetermined threshold, correspondence calculation section 161 caninclude the information, which indicates the absence of any combinationof an optimum correspondence in the other objects and object zones, intothe optimum correspondence information, provided that the differencebetween the degrees of combined similarity of the upper-ordercombinations is small (for example, equal to or smaller than apredetermined threshold) and besides, that there is no object and objectzone common to the upper-order combinations.

For the objects which the optimum-correspondence information indicatesas having no optimum corresponding relationship, correspondenceestablishment section 162 decides to designate the correspondingrelationship between objects and object zones, included in the firstzone-correspondence information, as the corresponding relationshipbetween objects and object zones.

As described above, for the combinations having the degrees of combinedsimilarity ranging from the highest degree of combined similarity towithin a predetermined threshold, of all the possible combinations,correspondence calculation section 161 selects, from the combinationsranked in the higher degrees of combined similarity, the combinationshaving the degrees of combined similarity within a prescribed threshold,includes the corresponding relationship between objects and object zonescommonly existing in the selected combinations, into theoptimum-correspondence information as optimum correspondences, and forthe objects and object zones that are not included in the correspondingrelationship between said commonly existing objects and object zones,correspondence calculation section 161 includes the information, whichindicates absence of an optimum correspondence between objects and anobject zone, into the optimum-correspondence information. It can beintended that, for the objects not indicated in theoptimum-correspondence information as having no optimum correspondencesto object zones, correspondence establishment section 162 provides theinformation indicative of the correspondences between objects and objectzones included in the optimum-correspondence information as thecorrespondence information that has been determined; and for the objectsthat are indicated in the optimum-correspondence information as havingno optimum correspondence to object zones, correspondence establishmentsection 162 provides the information indicative of the correspondingrelationship between objects and object zones included in the firstzone-correspondence information as the correspondence information thathas been determined.

Modified Embodiment 10

Further, in the above-described respective methods,characteristic-quantity generation section 14 can modify, depending onthe state of tracking, the information included in the zonecharacteristic-quantity information that is to be supplied tocorrespondence decision section 16. In this case, for example, for theobject zone that is indicated as being in the state of parting in itsstate of tracking by the first zone-correspondence information suppliedfrom state-of-tracking decision section 13, the characteristicquantities of the object zones of interest is included in the zonecharacteristic-quantity information. Alternatively, for the object zonesindicated as being in the state of tracking other than the state ofparting, the zone characteristic-quantity information need not include,while can include, the characteristic quantities of the object zones ofinterest, because correspondence decision section 16 does notnecessitate the characteristic quantities of the object zones. Stillalternatively, the zone characteristic-quantity information can includethe information indicating that there is no necessity of establishingcorrespondence, in place of including no characteristic quantities.Correspondence decision section 16 excludes the object zones indicatedin the zone characteristic-quantity information as not necessitateestablishing the correspondence to objects, from the establishment ofcorrespondence, thereby decreasing an amount of calculation ofcorrespondence decision section 16.

Modified Embodiment 11

Characteristic-quantity synthesis section 15 is not intended to acquirethe characteristic quantities of all the objects stored incharacteristic-quantity storage section 143 but is intended to acquirethe characteristic quantities of the objects stored incharacteristic-quantity storage section 143 only if the state oftracking of the objects is indicated as the state of parting in thefirst zone-correspondence information provided by state-of-trackingdecision section 13, whereby it is enabled to omit the redundantprocessing of acquiring the object characteristic quantities that arenot required to generate the synthesized characteristic quantities andto accelerate the processing of characteristic-quantity synthesissection 15 to generate the synthesized characteristic quantities.

Modified Embodiment 12

In the above-described methods, the first zone-correspondenceinformation is identical with the correspondence information that hasbeen determined for the objects in the states of tracking other than thestate of parting. Accordingly, the content of the information includedin the correspondence information that has been determined, fed back tostate-of-tracking decision section 13 from correspondence decisionsection 16 can be restricted only to the corresponding relationship ofthe objects in the state of parting. In this case, for the otherobjects, state-of-tracking decision section 13 employs, in place of thecorrespondence information that has been determined, the correspondingrelationship obtained by state-of-tracking decision section 13 itself inthe past (included in the first zone-correspondence information). Thisapproach as well enables realizing the device equivalent to theobject-tracking devices that use the above-described method.

Modified Embodiment 13

In each of the above-described methods, while characteristic-quantityupdate section 142 updates characteristic quantities of objects on thebasis of the first zone-correspondence information,characteristic-quantity update section 142 can update characteristicquantities of objects on the basis of the correspondence informationthat has been determined in place of the first zone-correspondenceinformation. In this case, characteristic-quantity extraction section141 includes the characteristic quantities of the object zones, whichare in the state of tracking indicated in the first zone-correspondenceinformation as being the stand-alone state or the state of parting, intothe zone characteristic-quantity information to be supplied tocharacteristic-quantity update section 142. Characteristic-quantityupdate section 142 decides the states of tracking of objects based onthe correspondence information that has been determined provided bycorrespondence decision section 16 after correspondence decision section16 determines the correspondences between objects and object zones.Characteristic-quantity update section 142 then, exclusively for theobjects being in the stand-alone state in their the state of tracking,updates the characteristic quantities of the objects stored incharacteristic-quantity storage section 143 based on thecharacteristic-quantity information about object zones provided bycharacteristic-quantity extraction section 141. This approach as wellallows realizing the device equivalent to the object-tracking deviceusing any of the above-described methods.

Modified Embodiment 14

Correspondence decision section 16 can decide the states of tracking ofobjects rather than characteristic-quantity update section 142 decidesthe states of tracking of objects on the basis of the correspondenceinformation that has been determined, wherein correspondence decisionsection 16 provides the correspondence information that has beendetermined that includes the information about the decided states oftracking of objects, thereby allowing characteristic-quantity updatesection 142 to omit the processing for deciding the states of trackingof the objects. This approach as well enables the realization of thedevice equivalent to the object-tracking device using any of theabove-described methods.

Modified Embodiment 15

Characteristic-quantity update section 142 can update the characteristicquantities of objects stored in characteristic-quantity storage section143 based on not exclusively the first zone-correspondence informationbut both the first zone-correspondence information and thecorrespondence information that has been determined. In this case,characteristic-quantity extraction section 141 includes thecharacteristic quantities of the object zones that are indicated in thefirst zone-correspondence information as having the stand-alone state orthe state of parting, into the zone characteristic-quantity informationto be provided to characteristic-quantity update section 142.Exclusively about the objects that are indicated as having thestand-alone state while in their tracking states,characteristic-quantity update section 142 updates the characteristicquantities of the objects stored in characteristic-quantity storagesection 143 on the basis of both the zone characteristic-quantityinformation supplied from characteristic-quantity extraction section 141and the first zone-correspondence information supplied fromstate-of-tracking decision section 13. Characteristic-quantity updatesection 142 decides the states of tracking of the objects based on thecorrespondence information that has been determined supplied fromcorrespondence decision section 16 after correspondence decision section16 decides the correspondences between objects and object zones, andexclusively about the objects of the state of tracking that has madetransition from the state of parting to the stand-alone state, updatesthe characteristic quantities of the objects stored incharacteristic-quantity storage section 143 based on the zonecharacteristic-quantity information supplied fromcharacteristic-quantity extraction section 141.

In this case as well, correspondence decision section 16 can decide thestates of tracking of objects rather than characteristic-quantity updatesection 142 decides the states of tracking of objects on the basis ofthe correspondence information that has been determined. Correspondencedecision section 16 further provides the correspondence information thathas been determined inclusive of the information about the decidedstates of tracking of the objects. This approach allowscharacteristic-quantity update section 142 to omit the processing ofdeciding the states of tracking of the objects.

(The Second Working Example of the Invention)

Explanation is next presented regarding the second working example ofthe invention with reference to the drawings. The firstzone-correspondence information in the present working example includesthe at-rest/in-motion information that indicates whether an object zoneis at rest or in motion. An at-rest/in-motion state includes an at-reststate and an in-motion state, wherein the at-rest state denotes thestate in which the object zone is at rest and the in-motion statedenotes the state in which the object zone is in motion. Theobject-tracking device treats the object zone that is in an at-reststate in its at-rest/in-motion state as a background.

The first zone-correspondence information includes background-updateinformation indicative of whether or not an object zone is treated as abackground. The background-update information is represented by meansof, for example, a binary representation of “0” and “1”: if thebackground-update information about an object zone is “1”, then theobject zone is treated as a background; and if the background-updateinformation about an object zone is “0”, then the object zone is nottreated as a background.

If the object-tracking device is intended for tracking a human body, forexample, the object-tracking device can take exclusively a human body asa tracking target by presuming static objects separated from the humanbody as a background. As a method of discriminating between a state ofat-rest and a state in-motion, a method can be employed, for example, inwhich, an object zone of the extracted object zones is decided as beingin a static state (an at-rest state) if the object zone keeps the staticstate for a duration time (hereinafter referred to as a static durationperiod) longer than a prescribed threshold, and the at-rest./in-motionstate of the object zone is decided to be an in-motion state if thestatic duration period is shorter than the prescribed threshold.

Through Introduction of background-update information, theobject-tracking device can deem the object zones, which have beenregarded as a background, as needless to track and exclude from thetracking target. In this way, the object-tracking device can trackexclusively the objects that are required to track as tracking targets.Specifically, it can be intended to exclude the combinations of objectsand object zones that are indicated by the at-rest/in-motion informationas taking the at-rest state, from all the possible combinations. Forexample, the object-tracking device, in treating object zones, deems theobject zones as a part of the background only if the object zones exceeda prescribed threshold in their static duration periods, keep theat-rest state in their at-rest/in-motion states and further are decidedin accordance with the correspondence information that has beendetermined as having no corresponding objects. In addition, the settingup of a threshold value for discriminating the background prevents theobject-tracking device from faultily deeming the object zone as abackground in the case where a target object exists, even when theobject-tracking device fails in tracking of the object.

FIG. 11 is a block diagram illustrating a second working example of theobject-tracking device according to the present invention. Although theconstituent elements of the object-tracking device are the same as withthe case of the first working example as FIG. 11 shows, object-zoneextraction section 12 extracts object zones on the basis of not only theimage information provided by way of image input terminal 11 but alsothe correspondence information that has been determined provided bycorrespondence establishment section 17. In the above device, theconstruction of correspondence establishment section 17 is the same asthat shown in FIG. 3, the construction of state-of-tracking decisionsection 13 is the same as that shown in FIG. 6, the construction ofcharacteristic-quantity generation section 14 is the same as that shownin FIG. 8 and the construction of correspondence decision section 16 isthe same as that shown in FIG. 9.

Object-zone-information storage section 133 stores static durationperiods of object zones. Object tracking section 131 associates the paststatic duration periods of object zones stored inobject-zone-information storage section 133 with the current objectzones and calculates the static duration periods at present.Object-zone-information storage section 133 stores the calculatedcurrent static duration periods of the object zones.

In calculating the latest static duration periods, a case may beenvisaged in which a plurality of the past static duration periods canbe associated with an object zone. When a plurality of past staticduration periods exist, object tracking section 131 selects the longestof the past static duration periods that are now in existence andassociates the selected static duration period with the current staticduration period. Alternatively, it is permissible to select the shorteststatic duration period to associate it with the current static durationperiod, or to calculate an average time of the static duration periodsto associate the calculated average time with the current staticduration period.

Object tracking section 131 calculates the current static durationperiods by performing the update of the static duration periodsaccording to the following procedures: object tracking section 131 findsthe difference vector between the center of gravity of a past objectzone and that of the latest object zone for each of the object zones; ifthe magnitude of difference vector is equal to or below a predeterminedthreshold, then object tracking section 131 decides that the object isat rest and updates the static duration period; and if the magnitude ofthe difference vector exceeds the predetermined threshold, then objecttracking section 131 decides that the object zone is in motion andresets the static duration period.

State decision section 132 decides the at-rest/in-motion states ofobject zones based on the static duration periods of object zones storedin object-zone-information storage section 133. State decision section132 provides the first zone-correspondence information inclusive ofstatic duration periods of object zones and the information about theat-rest/in-motion states. In addition, object-zone-information storagesection 133 updates the at-rest/in-motion states of object zones on thebasis of the decision of state decision section 132.

In the first embodiment, correspondence calculation section 161calculates the degrees of similarity between the characteristicquantities of object zones and synthesized characteristic quantities forall the combinations of the objects and object zones potentially to beassociated in corresponding relationship (all the possiblecombinations), involved in the combinations of the objects and objectzones. In the present embodiment, however, the calculation is made onlyof the combinations of the objects and object zones in which the objectzones at rest in their at-rest/in motion state are not associated withany objects in corresponding relationship. In other words, thecombinations between the object zones that are at rest in theirat-rest/in motion state and the corresponding objects are excluded fromall the possible combination, whereby correspondence calculation section161 can decrease the amount of calculation for calculating the degreesof similarity between objects and object zones.

Correspondence establishment section 162 provides the correspondenceinformation that has been determined including the information about thestatic duration periods of the object zones calculated by objecttracking section 131. Correspondence establishment section 162 decideswhether or not the object zone of interest is deemed as the backgroundbased on both the information about the static duration period and theinformation about the at-rest/in-motion state. Specifically, the objectzone is decided to deem as the background if static duration periodexceeds a predetermined threshold; the at-rest/in-motion state is anat-rest state; and there is no object associated in correspondingrelationship on the basis of the correspondence information that hasbeen determined. If an object zone is deemed as a background, thencorrespondence establishment section 162 incorporates thebackground-update information, which indicates the update of thebackground within a region of the object zone, into the correspondenceinformation that has been determined.

If the correspondence information that has been determined includes thebackground-update information indicative of updating the backgroundwithin a region of the object zone, then object-zone extraction section12 updates the image of the background within the region of the objectzone of interest on the basis of the object-zone information, the imageinformation supplied by way of image input terminal 11 and thecorrespondence information that has been determined provided bycorrespondence establishment section 17. Object-zone extraction section12 updates the image of the background according to formula (13).B _(t)(x,y)−(1−μ)B _(t−1)(x,y)+μI _(in)(0≦μ≦1)  (13)where (x, y) denotes the coordinates of a pixel and I_(t0)(x, y) standsfor the pixel value of the image information at coordinates (x, y).B_(t)(x, y) and B_(t−i)(x, y) denote the pixel values of the backgroundimages at coordinates (x, y) at times t, and t−1, respectively. Further,μ represents an update coefficient.

As described above, the discrimination between an object and thebackground is enabled by providing the first zone-correspondenceinformation that includes the static duration period andat-rest/in-motion state of an object zone, from state decision section132 and by providing correspondence information that has been determinedthat includes the background-update information, from correspondenceestablishment section 17. Furthermore, the inclusion of theat-rest/in-motion state information in the first zone-correspondenceinformation allows the object-tracking device to exclude the objects atrest from the candidate objects to be brought into correspondingrelationship to the object zones of interest, whereby it is enabled thatthe amount of the calculation to be carried out by correspondencecalculation section 161 for calculating the degrees of similarity isdecreased, and furthermore, only objects in motion are tracked.

Alternatively, object tracking section 131 can provide the secondzone-correspondence information inclusive of the information aboutstatic duration periods to state decision section 132, rather thanobject-zone-information storage section 133 stores static durationperiods and state decision section 132 acquires the static durationperiods stored in object-zone-information storage section 133. Thisstrategy allows state decision section 132 to acquire the informationabout static duration periods without any intervention ofobject-zone-information storage section 133.

For supplement, for the object zones that are indicated as being in thestate of parting in their state of tracking and also indicated as beingat rest in their at-rest/in-motion state, characteristic-quantityextraction section 141 may include but need not include anycharacteristic quantities of the object zones in the zonecharacteristic-quantity information, because correspondence calculationsection 161 does not need the characteristic quantities of the objectzones. Alternatively, the zone characteristic-quantity information caninclude the information indicating that there is no need for anycorrespondence establishment, in place of including no characteristicquantities of the object zones. In this way, the redundancy of providingunnecessary characteristic quantities can be saved.

(The Third Embodiment of the Invention)

Explanation is next presented regarding a third embodiment of thepresent invention referring to drawings. FIG. 12 is a block diagramillustrating a third working example of the object-tracking device ofthe present invention. As illustrated in FIG. 12, the object-trackingdevice comprises image input terminal 21, first control section 22,object-zone extraction section 23, characteristic-quantity generationsection 24 and characteristic-quantity synthesis section 25.

First control section 22 is connected to image input terminal 21,object-zone extraction section 23, characteristic-quantity generationsection 24 and characteristic-quantity synthesis section 25 to controlthese sections. First control section 22, in addition, performs thecorrespondence establishment between objects and object zones(associates objects and object zones in corresponding relationship) andprovides the correspondence information that has been determined.

Object-zone extraction section 23 extracts object zones on the basis ofthe image information provided from first control section 22 andsupplies object-zone information that includes the image informationabout object zones numbered with zone numbers, to first control section22.

Characteristic-quantity generation section 24 extracts thecharacteristic quantities of object zones on the basis of the imageinformation and the object-zone information both supplied from firstcontrol section 22. Characteristic-quantity generation section 24,furthermore, supplies the zone characteristic-quantity informationinclusive of the extracted characteristic quantities of the objectzones, to first control section 22.

Characteristic-quantity synthesis section 25 calculates synthesizedcharacteristic quantities of objects that are generated by synthesizingthe characteristic quantities of a plurality of objects for all requiredcombinations of the plurality of objects, on the basis of the zonecharacteristic-quantity information and the first zone-correspondenceinformation supplied from first control section 22, and thencharacteristic-quantity synthesis section 25 supplies the synthesizedcharacteristic-quantity information inclusive of the synthesizedcharacteristic quantities to first control section 22.

Like the first working example, the first first zone-correspondenceinformation includes the information about the correspondingrelationship between objects and object zones as well as the informationabout the states of tracking of the objects. In addition, the firstzone-correspondence information is generated at first control section 22in the present embodiment.

First control section 22 receives image information by way of imageinput terminal 21 for receiving image signals provided from a videocamera or the like. First control section 22 supplies the imageinformation to object-zone extraction section 23 and receives theobject-zone information from object-zone extraction section 23. Firstcontrol section 22 further generates the first zone-correspondenceinformation on the basis of the object-zone information and the pastcorrespondence information that has been determined and supplies thegenerated first zone-correspondence information, the image informationand the object-zone information to characteristic-quantity generationsection 24. First control section 22, still further, receives the zonecharacteristic-quantity information supplied fromcharacteristic-quantity generation section 24, and provides the zonecharacteristic-quantity information and the first zone-correspondenceinformation to characteristic-quantity synthesis section 25.Furthermore, first control section 22 receives the synthesizedcharacteristic-quantity information supplied fromcharacteristic-quantity synthesis section 25.

First control section 22 calculates the degrees of similarity betweenthe characteristic quantities of the object zones included in the zonecharacteristic-quantity information and the synthesized characteristicquantities included in the synthesized characteristic-quantityinformation, calculates the degrees of combined similarity and decidesthe optimum correspondences between the objects and object zones. Firstcontrol section 22 then provides the information about the decidedcorresponding relationship between the objects and object zones ascorrespondence information that has been determined.

FIG. 13 is a block diagram illustrating an example of the constructionof first control section 22. As shown in FIG. 13, first control section22 includes second control section 221, object-tracking section 222,state-deciding section 223 and correspondence-calculation section 224.

It is to be noted that, while first control section 22 (second controlsection 221, object-tracking section 222, state-deciding section 223 andcorrespondence-calculation section 224), object-zone extraction section23, characteristic-quantity generation section 24 andcharacteristic-quantity synthesis section 25 can be realized byhardware, they can be realized by software as well. Specifically, thesesections can be realized through the use of a CPU, which is capable ofexecuting the processes to be performed by first control section 22,object-zone extraction section 23, characteristic-quantity generationsection 24 and characteristic-quantity synthesis section 25 inaccordance with programs and also the program stored in a storage devicefor implementing the functions of first control section 22, object-zoneextraction section 23, characteristic-quantity generation section 24 andcharacteristic-quantity synthesis section 25, as explained below.

Second control section 221, upon receiving image information by way ofimage input terminal 21, provides the image information to object-zoneextraction section 23. Second control section 221 further supplies thecurrent object-zone information provided by object-zone extractionsection 23, the past object-zone information stored by second controlsection 221 itself and the past correspondence information that has beendetermined stored in second control section 221 itself toobject-tracking section 222. Second control section 221 further suppliesthe image information received through image input terminal 21 and theobject-zone information provided by object-zone extraction section 23 tocharacteristic-quantity generation section 24. The content of thecorrespondence information that has been determined is the same as thatof the first working example.

Second control section 221, upon receiving the secondzone-correspondence information from object-tracking section 222,supplies the second zone-correspondence information and the object-zoneinformation to state-deciding section 223. The content of the secondzone-correspondence information is the same as that of the first workingexample.

Still further, second control section 221, upon receiving the firstzone-correspondence information from state-deciding section 223,supplies the characteristic quantities of objects and the firstzone-correspondence information to characteristic-quantity synthesissection 25 only in regard to the object zones that are indicated in thefirst zone-correspondence information as being in the state of partingwhile in their tracking states. The content of the firstzone-correspondence information is the same as that of the first workingexample.

Second control section 221, upon receiving synthesizedcharacteristic-quantity information from characteristic-quantitysynthesis section 25, provides the zone characteristic-quantityinformation, the synthesized characteristic-quantity information and thefirst zone-correspondence information to correspondence-calculationsection 224. Further, for the objects that are indicated in the firstzone-correspondence information supplied from state-deciding section223, as being non-parting (in the state other than the state of parting)while in their tracking states, second control section 221 decides todesignate the correspondences to the object zones included in the firstzone-correspondence information as the corresponding relationshipbetween the objects of interest and the object zones. For the objectsthat are indicated in the first zone-correspondence information as beingparting while in their tracking states, second control section 221decides to designate the correspondences to the object zones included inthe optimum-correspondence information provided bycorrespondence-calculation section 224 as the corresponding relationshipbetween the objects of interest and the object zones. The content of theoptimum-correspondence information provided bycorrespondence-calculation section 224 is the same as that described inthe first working example.

Object-tracking section 222 tracks objects based on the currentobject-zone information received from second control section 221, thepast object-zone information stored in second control section 221 andthe past correspondence information that has been determined andsupplies the second zone-correspondence information to second controlsection 221. The method by which object-tracking section 222 tracks anobject is the same as the method of tracking an object in the firstworking example.

State-deciding section 223 decides the states of tracking of objectsbased on the second zone-correspondence information and the object-zoneinformation both received from second control section 221 and suppliesthe first zone-correspondence information to second control section 221.

Correspondence-calculation section 224, based on the zonecharacteristic-quantity information provided from second control section221, the synthesized characteristic-quantity information and the firstzone-correspondence information, calculates the degrees of similaritybetween the synthesized characteristic quantities of objects andcharacteristic quantities of object zones for all the possiblecombinations of objects and object zones and calculates the degrees ofcombined similarity; decides to designate the combination of the objectsand object zones having the highest degree of combined similarity as anoptimum correspondence between the objects and object zones and providesthe optimum correspondence information; and supplies the optimumcorrespondence information to second control section 221.

Explanation next regards the operation of the object-tracking deviceaccording to the third embodiment. FIG. 14 is a flow chart illustratingan example of the process of the object-tracking device. When receivingimage information by way of image input terminal 21 (Step S1401),object-zone extraction section 23 extracts object zones from the imageinformation (Step S1402) and supplies object-zone information includingthe image information about the object zones.

Object-tracking section 222 tracks an object and associates the objectand object zone in corresponding relationship to provide the secondzone-correspondence information. State-deciding section 223 decides thestates of tracking of the object based on the second zone-correspondenceinformation and the object-zone information (Step S1403).Characteristic-quantity generation section 24 calculates thecharacteristic quantities of object zones (Step S1404).

Second control section 221 is provided with counter i. Second controlsection 221 sets the number of objects on counter i (Step S1405). Secondcontrol section 221 decides whether or not the value of counter i is “0”(Step S1406) and if the value of counter i is “0”, then ends theprocess. If the value of counter i is not “0”, then second controlsection 221 instructs the execution of the next step S1407 tostate-deciding section 223.

State-deciding section 223 decides whether or not the state of trackingan object is the state of parting (Step S1407). If state-decidingsection 223 decides that the state of tracking of the object is a stateof parting, then characteristic-quantity synthesis section 25synthesizes characteristic quantities of objects for all the requiredcombinations of a plurality of objects to generate synthesizedcharacteristic quantities (Step S1408).

Correspondence-calculation section 224 calculates the degrees ofsimilarity between synthesized characteristic quantities of objects andcharacteristic quantities of object zones and calculates the degrees ofcombination similarities, for all of the possible combinations ofobjects and object zones. Second control section 221 decides todesignate the combination of objects and object zones that has thehighest degree of similarity calculated by correspondence-calculationsection 224 as an optimum correspondence relation between objects andobject zones (Step S1409).

If state-deciding section 223 decides in Step S1407 that the state oftracking is a state other than the state of parting, then second controlsection 221 decides to designate the correspondences between objects andobject zones included in the first zone-correspondence information asoptimum corresponding relationship between objects and object zones(Step S1409).

Characteristic-quantity generation section 24 updates the characteristicquantities of the objects that are decided on the correspondingrelationship (Step S1410), wherein characteristic-quantity generationsection 24 updates exclusively the characteristic quantities of theobjects that are indicated as being stand-alone while in their trackingstates.

Second control section 221 next decrements the counter value by 1 (StepS1411), and the processes are repeatedly executed from Step S1407 toStep S1410 until the counter value is decided to be “0” at Step S1406.In other words, the processes from Step S1407 to Step S1410 are executeduntil optimum corresponding relationship to object zones are decided forall the objects. Upon completing the processes (from Step S1407 to StepS1410) to determine the corresponding relationship to object zones forall the objects, the processes are again executed from Step S1401.

The construction of the object-tracking device as described aboveenables realizing the object-tracking device having the same function asthe first working example.

It is to be noted that the constructions illustrated in FIGS. 12 and 13can be realized through the use of software. Specifically, in thepresent embodiment, the object-tracking device can be realized by meansof the object-tracking program for executing the following processes:image input process for receiving image information; state-of-trackingdecision process for deciding the state of tracking of each object orobject zone on the basis of the object-zone information and thecorrespondence information that has been determined indicative of thecorresponding relationship between object zones and objects prior to thepresent and providing the first zone-correspondence information thatindicates the corresponding relationship between the object zones andobjects and the states of tracking; characteristic-quantity generationprocess for generating zone characteristic quantities that representcharacteristic quantities of object zones and object characteristicquantities that represent characteristic quantities of objects throughthe use of the image information, the object zone information and thedecision result of the state-of-tracking decision means; andcorrespondence establishment process to determine the correspondingrelationship between objects and object zones through deciding todesignate the correspondences between objects and object zones includedin the first zone-correspondence information as the correspondencesbetween the objects and object zones if the objects are decided to be inthe states other than the state of parting while in their trackingstates, the state of parting being transient state through which anobject zone is parting into a plurality of object zones, andalternatively if the objects are decided to be in the states of partingwhile in their tracking states, synthesize characteristic quantities forall the required combinations of a plurality of objects based on thecharacteristic quantities of objects and the first zone-correspondenceinformation and generate each of synthesized characteristic quantities,compare each synthesized characteristic quantity with each zonecharacteristic quantity, and associate the objects and object zones incorresponding relationship under condition that the synthesizedcharacteristic quantities of the objects and zone characteristicquantities of the object zones are of the highest degree of combinedsimilarity.

1. An object-tracking device for tracking an object based on imageinformation, comprising: a characteristic-quantity synthesizing meansadapted to synthesize characteristic quantities of objectsrepresentative of characteristic quantities of respective objectsincluded in said image information for generating synthesizedcharacteristic quantities; and a correspondence-establishing means forestablishing correspondences between object zones and objects on thebasis of degrees of similarity between characteristic quantities of saidobject zones and said synthesized characteristic quantities, whereinsaid object zones refer to the zones that are extracted from said imageinformation and include the objects of interest.
 2. An object-trackingdevice according to claim 1, wherein said characteristic-quantitysynthesizing means is adapted to synthesize characteristic quantitiesfor each of all required combinations of a plurality of objects togenerate said synthesized characteristic quantities, and saidcorrespondence-establishing means establishes correspondences betweenobjects and object zones through comparing each of said synthesizedcharacteristic quantities generated by said characteristic-quantitysynthesizing means and zone characteristic quantities representative ofthe characteristic quantities of object zones.
 3. An object-trackingdevice according to claim 2, provided with: an object-zone extractingmeans for extracting said object zones from said image information andproviding the object-zone information that includes the imageinformation about said object zones, a state-of-tracking deciding meansfor deciding the states of tracking of individual objects or objectzones, wherein said state-of-tracking means relative positions of eachobject with respect to other objects, and a characteristic-quantitygenerating means for generating said zone characteristic quantities andobject characteristic quantities through the use of said imageinformation, said object-zone information and the decision resultseffected by said state-of-tracking deciding means, wherein saidcharacteristic-quantity synthesizing means generates synthesizedcharacteristic quantities through the use of said object characteristicquantities and the decision results effected by said state-of-trackingdeciding means.
 4. An object-tracking device according to claim 3,wherein said state-of-tracking deciding means decides the states oftracking of respective objects or object zones based on the object-zoneinformation and the correspondence information that has been determinedthat indicates the corresponding relationship between the object zonesand objects prior to the present to provide the firstzone-correspondence information that indicates the correspondingrelationship among the object zones and objects and said states oftracking, said characteristic-quantity generating means generates zonecharacteristic quantities and object characteristic quantities based onthe current image information, said object-zone information, said firstzone-correspondence information and said correspondence information thathas been determined, said characteristic-quantity synthesizing meansgenerates synthesized characteristic quantities that serve as candidatesto be placed in the corresponding relationship to individual objectzones based on said object characteristic quantities and said firstzone-correspondence information to provide synthesizedcharacteristic-quantity information, wherein said synthesizedcharacteristic-quantity information is the information that includessynthesized characteristic quantities and the corresponding relationshipbetween the synthesized characteristic quantities and objects used forthe generation of said synthesized characteristic quantities, and saidcorrespondence-establishing means includes a correspondence-determiningmeans that associates objects and object zones to place in thecorresponding relationship based on said first zone-correspondenceinformation, zone characteristic-quantity information that is theinformation indicative of said zone characteristic quantities and saidsynthesized characteristic-quantity information to provide saidcorrespondence information that has been determined in the present time5. An object-tracking device according to claim 3, wherein said state oftracking includes at least one of or a combination of: a stand-alonestate in which only a single object resides in an object zone; acrossover state in which a plurality of objects correspond to a singleobject zone; and a state of parting that is a transient state in which asingle object zone is parted into a plurality of object zones.
 6. Anobject-tracking device according to claim 4, wherein said state oftracking includes at least one of or a combination of: a stand-alonestate in which only a single object resides in an object zone; acrossover state in which a plurality of objects correspond to a singleobject zone; and a state of parting that is a transient state in which asingle object zone is parted into a plurality of object zones.
 7. Anobject-tracking device according to claim 4, wherein saidcharacteristic-quantity generating means generates zone characteristicquantities, each including at least one of or one of combinations of acolor histogram, area, image template and color histogram normalizedwith respect to said area, of the object zone, and finds an object zonecorresponding to the object of interest from the firstzone-correspondence information and provides at least one or one ofcombinations of a color histogram, area, image template and colorhistogram normalized with respect to said area of the object zone as anobject characteristic quantity.
 8. An object-tracking device accordingto claim 4, wherein said state-of-tracking deciding means includes anobject-zone storing means for storing the object-zone information, anobject-tracking means for tracking an object based on said object-zoneinformation, the correspondence information that has been determined andthe object-zone information prior to the present that is provided fromsaid object-zone storing means and further providing a secondzone-correspondence information that indicates the correspondencesbetween objects and object zones, and a state-deciding means fordeciding the states of tracking of objects based on said secondzone-correspondence information, said object-zone information and saidobject-zone information prior to the present and providing said firstzone-correspondence information.
 9. An object-tracking device accordingto claim 8, wherein said state-deciding means, based on at least one ofor one of the combinations of the correspondences between objects andobject zones, distances between object zones and continued periods ofseparation of said object zones, obtained from said secondzone-correspondence information and object-zone information, groups theobjects that have a common region in their corresponding object zones tosort the objects and corresponding object zones into one class, andsorts the object, which differs in the corresponding object zone fromany other objects, and the object zone corresponding thereto into oneclass to sort the objects and object zones into a plurality of classes,and decides the state of tracking on the basis of the sorted classes.10. An object-tracking device according to claim 9, wherein said stateof tracking includes the state of parting that is a transient statethrough which an object zone parts into a plurality of object zones,said state-deciding means decides that, if two or more object zones areincluded in a sorted class, then the class meets the condition of beingin a state of parting, and that, if a class meets the condition of beingin a state of parting, the states of tracking of the objects and objectzones included in the class are the state of parting.
 11. Anobject-tracking device according to claim 10, wherein if the sortedclass meets the condition of being in the state of parting and if thesorted class meets at least one of or one of the combinations of theconditions that two or more objects are included in said class, thateach of the distances between the object zones included in said classexceeds a predetermined threshold and that continued periods ofseparation of the object zones included in said class exceed apredetermined threshold, said state-deciding means decides that thestates of tracking of the objects and object zones included in the classare said state of parting.
 12. An object-tracking device according toclaim 10, wherein said state of tracking includes a state of parting anda stand-alone state in which a single object resides in an object zone,and if the sorted class includes only one object and if the states oftracking of the object and the object zone included in said class arenot the state of parting, then said state-deciding means decides thatthe states of tracking of the object and the object zone included insaid class are the stand-alone state.
 13. An object-tracking deviceaccording to claim 11, wherein said state of tracking includes a stateof parting and a stand-alone state in which a single object resides inan object zone, and if the sorted class includes only one object and ifthe states of tracking of the object and the object zone included insaid class are not the state of parting, then said state-deciding meansdecides that the states of tracking of the object and the object zoneincluded in said class are the stand-alone state.
 14. An object-trackingdevice according to claim 10, wherein said state of tracking includes astate of parting and also a crossover state in which a plurality ofobjects are in corresponding relationship to a single object zone, andif a sorted class includes two or more objects and if the states oftracking of the objects and the object zones included in said class arenot the state of parting, said state-deciding means decides that thestates of tracking of the objects and the object zones included in saidclass are the crossover state.
 15. An object-tracking device accordingto claim 11, wherein said state of tracking includes a state of partingand also a crossover state in which a plurality of objects are incorrespondence relation to a single object zone, and if a sorted classincludes two or more objects and if the states of tracking of theobjects and the object zones included in said class are not the state ofparting, said state-deciding means decides that the states of trackingof the objects and the object zones included in said class are thecrossover state.
 16. An object-tracking device according to claim 4,wherein said characteristic-quantity generating means includes: acharacteristic-quantity extracting means for extracting zonecharacteristic quantities from the image information, object-zoneinformation and the first zone-correspondence information and providingthe zone characteristic-quantity information that is the informationindicative of said zone characteristic quantities;characteristic-quantity storing means for storing object characteristicquantities and selecting the stored object characteristic quantities tosupply the selected object characteristic quantities, as required, andcharacteristic-quantity updating means for updating said objectcharacteristic quantities stored in said characteristic-quantity storingmeans based on said zone characteristic-quantity information, said firstzone-correspondence information or correspondence information that hasbeen determined and the object characteristic quantities generated priorto the present.
 17. An object-tracking device according to claim 16,wherein said state of tracking includes the state of parting that is atransient state through which an object zone parts into a plurality ofobject zones, and said characteristic-quantity extracting meansincludes, in zone characteristic-quantity information, the informationindicating that there is no need for establishing correspondences toobjects for the object zones that represent the states other than thestate of parting while in their tracking states, and saidcorrespondence-determining means excludes, from the establishment of thecorresponding relationship, the object zones indicated in said zonecharacteristic-quantity information as there is no need to establishcorresponding relationship to objects.
 18. An object-tracking deviceaccording to claim 16, wherein said state of tracking includes astand-alone state in which a single object resides in an object zone,and said characteristic-quantity updating means decides whether or notthe state of tracking of an object is the stand-alone state on the basisof the first zone-correspondence information or the correspondenceinformation that has been determined, and if the state of tracking ofthe object is the state other than the stand-alone state, does notupdate the object characteristic quantities stored in saidcharacteristic-quantity storing means.
 19. An object-tracking deviceaccording to claim 4, wherein said characteristic-quantity synthesizingmeans determines all possible combinations of objects and object zonesbased on the object characteristic quantities generated by saidcharacteristic-quantity generating means and the firstzone-correspondence information, and synthesizes object characteristicquantities only for the determined combinations of objects and objectzones to generate synthesized characteristic quantities.
 20. Anobject-tracking device according to claim 4, whereincharacteristic-quantity synthesizing means calculates the synthesisratios that are coefficients for adjusting the ratios at which theobject characteristic quantities are synthesized, and generatessynthesized characteristic quantities on the basis of said synthesisratios and object characteristic quantities.
 21. An object-trackingdevice according to claim 4, wherein said characteristic-quantitysynthesizing means receives zone characteristic quantities as well asobject characteristic quantities from the characteristic-quantitygenerating means, calculates synthesized characteristic quantitiesdepending on desired synthesis ratios on the basis of the received zonecharacteristic-quantity information and object characteristicquantities, and provides the synthesized characteristic quantity for thesynthesis ratio that yields the highest of all degrees of similaritybetween the calculated synthesized characteristic quantities and thezone characteristic quantities.
 22. An object-tracking device accordingto claim 4, wherein said state of tracking includes a state of partingthat is a transient state through which an object zone parts into aplurality of object zones, and said characteristic-quantity synthesizingmeans generates synthesized characteristic quantities only for theobject zones that are indicated as having the state of parting as theirstates of tracking.
 23. An object-tracking device according to claim 4,wherein said object characteristic quantity includes an area of anobject, and said characteristic-quantity synthesizing means calculatesthe synthesis ratios that are coefficients for adjusting the ratios atwhich the object characteristic quantities are synthesized on the basisof the areas of objects included in said object characteristicquantities and generates synthesized characteristic quantities from saidsynthesis ratios and said object characteristic quantities.
 24. Anobject-tracking device according to claim 23, wherein saidcharacteristic-quantity synthesizing means limits the synthesis ratioswithin a predetermined range on the basis of the variations in the areasof objects.
 25. An object-tracking device according to claim 4, whereinsaid characteristic-quantity synthesizing means receives zonecharacteristic quantities together with object characteristic quantitiesfrom the characteristic-quantity generating means, calculatessynthesized characteristic quantities within the range of the variationsin the areas of objects based on the received zone characteristicquantities and object characteristic quantities, and provides thesynthesized characteristic quantities that have the highest degrees ofsimilarity to the zone characteristic quantities of the object zones ofinterest.
 26. An object-tracking device according to claim 4, whereinsaid object characteristic quantity includes an image templaterepresentative of the shape and color of an object, and saidcharacteristic-quantity synthesizing means decides the back-to-bellyrelation of each of the objects from the image templates and zonecharacteristic quantities and obtains the synthesized characteristicquantities by synthesizing the image templates based on the respectivedecided back-to-belly relations of said objects.
 27. An object-trackingdevice according to claim 4, wherein said correspondence-determiningmeans is provided with a correspondence-calculating means forcalculating the combination of objects and object zones which have thehighest similarity from all the possible combinations of the objects andobject zones that are possibly associated in corresponding relationshipbased on said synthesized characteristic-quantity information, said zonecharacteristic-quantity information and said first zone-correspondenceinformation, selecting the calculated combination of objects and objectzones as an optimum combination and generating theoptimum-correspondence information that indicates the optimumcorresponding relationship between objects and object zones, and acorrespondence-deciding means for determining the correspondingrelationship between objects and object zones on the basis of said firstzone-correspondence information and said optimum-correspondenceinformation and providing the correspondence information that has beendetermined that is the information that includes the correspondingrelationship that has been decided between objects and object zones. 28.An object-tracking device according to claim 27, wherein saidcorrespondence-calculating means calculates a total degree of similarityfor each of all the possible combinations of objects and object zones,said total degree of similarity being a sum of the degrees of similaritybetween the characteristic quantities of object zones and synthesizedcharacteristic quantities within each combination, and decides thecombination that has the highest similarity based on the combinationhaving the highest, total degree of similarity, of said all the possiblecombinations.
 29. An object-tracking device according to claim 27,wherein said first zone-correspondence information includes theinformation about an at-rest/in-motion state that indicates whether anobject zone is at rest or in motion, and said correspondence-calculatingmeans excludes the combination of the object and object zone that isindicated as being at rest in said information about anat-rest/in-motion state from said all possible combinations.
 30. Anobject-tracking device according to claim 27, wherein if the degrees ofcombined similarity that can be obtained from the degrees of similarityof the sets of the objects and object zones, said sets of the objectsand object zones making up the combinations decided to be ranked as thehighest similarity, are equal to or lower than a predeterminedthreshold, then said correspondence-calculating means selects thecombinations of the degrees of combined similarity within saidpredetermined threshold, from the combinations of the degrees ofcombined similarity ranked as the highest similarity of all possiblecombinations of objects and object zones, includes the correspondingrelationship of objects and object zones common to the selectedcombinations, into the optimum-correspondence information as optimumcorrespondences, and further, for the objects and object zones havingthe corresponding relationship that are not included in saidcorresponding relationship of the object and object zone common to saidselected combinations, includes the information indicating that thereare no optimum correspondence between the objects and object zones, intothe optimum-correspondence information, for the objects not indicated ashaving no optimum corresponding relationship to any object zones in saidoptimum-correspondence information, said correspondence-deciding meansprovides the information indicating the corresponding relationship ofobjects and object zones included in said optimum-correspondenceinformation as the correspondence information that has been determined,and for the objects indicated as having no optimum correspondingrelationship to any object zones in said optimum-correspondenceinformation, said correspondence-deciding means provides the informationindicating the corresponding relationship of objects and object zonesincluded in said first zone-correspondence information as thecorrespondence information that has been determined.
 31. Anobject-tracking device according to claim 28, wherein if the degrees ofcombined similarity that can be obtained from the degrees of similarityof the sets of the objects and object zones, said sets of the objectsand object zones making up the combinations decided to be ranked as thehighest similarity, are equal to or lower than a predeterminedthreshold, then said correspondence-calculating means selects thecombinations of the degrees of combined similarity within saidpredetermined threshold, from the combinations of the degrees ofcombined similarity ranked as the highest similarity of all possiblecombinations of objects and object zones, includes the correspondingrelationship of objects and object zones common to the selectedcombinations, into the optimum-correspondence information as optimumcorrespondences, and further, for the objects and object zones havingthe corresponding relationship that are not included in saidcorresponding relationship of the object and object zone common to saidselected combinations, includes the information indicating that thereare no optimum correspondence between the objects and object zones, intothe optimum-correspondence information, for the objects not indicated ashaving no optimum corresponding relationship to any object zones in saidoptimum-correspondence information, said correspondence-deciding meansprovides the information indicating the corresponding relationship ofobjects and object zones included in said optimum-correspondenceinformation as the correspondence information that has been determined,and for the objects indicated as having no optimum correspondingrelationship to any object zones in said optimum-correspondenceinformation, said correspondence-deciding means provides the informationindicating the corresponding relationship of objects and object zonesincluded in said first zone-correspondence information as thecorrespondence information that has been determined.
 32. Anobject-tracking device according to claim 27, wherein said state oftracking includes a state of parting that is a transient state throughwhich an object zone parts into a plurality of object zones, and saidcorrespondence-deciding means determines the corresponding relationshipbetween objects and object zones to be indicated in theoptimum-correspondence information only for the object zones thatexhibit a state of parting as their states of tracking.
 33. Anobject-tracking device according to claim 27, wherein said state oftracking includes a state of parting that is a transient state throughwhich an object zone parts into a plurality of object zones, and saidcorrespondence-deciding means provides the correspondences betweenobjects and object zones included in the first zone-correspondenceinformation as the correspondence information that has been determinedonly for the object zones that exhibit states other than the state ofparting while in their tracking states.
 34. An object-tracking methodfor tracking an object based on image information, comprising steps of:synthesizing object characteristic quantities, which representcharacteristic quantities of respective objects included in said imageinformation, to generate a synthesized characteristic quantity, andestablishing corresponding relationship between object or objects andobject zone on the basis of the degree of similarity between saidsynthesized characteristic quantity and characteristic quantity of saidobject zone, wherein said object zone is a region extracted from saidimage information and including said object or objects.
 35. Anobject-tracking method according to claim 34, including steps ofsynthesizing characteristic quantities for each of all requiredcombination of a plurality of objects to generate a synthesizedcharacteristic quantity, and establishing corresponding relationshipbetween object or objects and object zone through comparison of thegenerated synthesized characteristic quantity and zone characteristicquantity that represents the characteristic quantity of said objectzone.
 36. An object-tracking method according to claim 35, includingsteps of extracting an object zone from said image information andproviding the object-zone information that includes image informationabout said object zone, deciding a state of tracking representative of arelative position with respect to another object for every object orobject zone, generating zone characteristic quantities, objectcharacteristic quantities through the use of said image information,said object-zone information, and the decision results, and generatingsynthesized characteristic quantities through the use of said objectcharacteristic quantities and said decision results.
 37. Anobject-tracking method according to claim 36, including steps ofdeciding the state of tracking of every object or every object zonebased on said object-zone information and the correspondence informationthat has been determined that indicates the corresponding relationshipbetween object zones and objects prior to the present and providing thefirst zone-correspondence information indicative of the correspondingrelationship between the objects and object zones and the states oftracking, generating zone characteristic quantities and objectcharacteristic quantities based on the present image information, saidobject-zone information, said first zone-correspondence information andsaid correspondence information that has been determined, generating asynthesized characteristic quantity that functions as a candidate to beplaced in corresponding relationship to each object zone on the basis ofsaid object characteristic quantities and said first zone-correspondenceinformation and providing the synthesized characteristic-quantityinformation, which is the information that includes said synthesizedcharacteristic quantities and the corresponding relationship betweensynthesized characteristic quantities and objects used for generatingthe synthesized characteristic quantities of interest, and establishingcorrespondences between objects and object zones based on said firstzone-correspondence information, zone characteristic-quantityinformation, which is the information indicative of said zonecharacteristic quantities, and said synthesized characteristic-quantityinformation, and providing said correspondence information that has beendetermined at present.
 38. An object-tracking method according to claim36, wherein said state of tracking includes at least one of or one ofcombinations of a stand-alone state, in which only a single object ispresent in an object zone, a crossover state, in which a plurality ofobjects are present in a single object zone, and a state of parting thatis a transient state through which an object zone parts into a pluralityof object zones.
 39. An object-tracking method according to claim 37,wherein said state of tracking includes at least one of, or one ofcombinations of a stand-alone state, in which only a single object ispresent in an object zone, a crossover state, in which a plurality ofobjects are present in a single object, and a state of parting that is atransient state through which an object zone parts into a plurality ofobject zones.
 40. An object-tracking method according to claim 37,including steps of generating, as a zone characteristic quantity, atleast one of, or one of the combinations of the color histograms, areas,image templates and color histograms normalized with respect torespective areas, of object zones, and seeking the object zonescorresponding to said objects from said first zone-correspondenceinformation, and generating, as an object characteristic quantity, atleast one of, or one of the combinations of the color histograms, areas,image templates and color histograms normalized with respect torespective areas of said object zones.
 41. An object-tracking methodaccording to claim 37, including steps of storing said object-zoneinformation, tracking an object on the basis of said object-zoneinformation, the correspondence information that has been determined andthe object-zone information prior to the present and providing a secondzone-correspondence information indicative of the correspondence betweenthe object and object zone, and deciding the state of tracking an objecton the basis of said second zone-correspondence information, saidobject-zone information and said object-zone information prior to thepresent and providing said first zone-correspondence information.
 42. Anobject-tracking method according to claim 41, including steps of sortingobjects and object zones into a plurality of classes by: grouping theobjects based on those objects which have a common region in thecorresponding object zone, to enroll said objects and the correspondingobject zones in one class; and for the objects that correspond to theobject zones that differ from the object zones corresponding to anyother objects, enrolling the objects and the corresponding object zonein one class, based on at least one of, or one of combinations of thecorresponding relationship between objects and object zones, thedistances between object zones, and the duration period for on-partingof object zones calculated from said second zone-correspondenceinformation and said object-zone information; and deciding the state oftracking based on the classified class.
 43. An object-tracking methodaccording to claim 42, wherein said state of tracking includes a stateof parting that is a transient state through which a single object zoneparts into a plurality of object zones, and said object-tracking methodincludes steps of deciding that a classified class meets the conditionof being in the state of parting if the class includes two or moreobject zones, and if a class meets the condition of being in the stateof parting, deciding the state of parting based on the states oftracking the objects and object zones included in the class of interest.44. An object-tracking method according to claim 43, including a stepof: if the classified class meets the condition of being in the state ofparting and further meets at least one of, or one combination of theconditions that said class includes two or more objects, that thedistances between the object zones included in said class exceed apredetermined threshold, and that the continued periods of separation ofthe object zones included in said class exceed a predeterminedthreshold, deciding the state of parting based on the states of trackingthe objects and object zones included in said class.
 45. Anobject-tracking method according to claim 43, wherein said state oftracking includes the state of parting and a stand-alone state in whicha single object is present in an object zone, and said object-trackingmethod includes a step of deciding that the states of tracking of theobject and object zone are the stand-alone state if the classified classincludes a single object and also neither of the states of tracking ofthe object and the object zone included in the class is the state ofparting.
 46. An object-tracking method according to claim 44, whereinsaid state of tracking includes the state of parting and a stand-alonestate in which a single object is present in an object zone, and saidobject-tracking method includes a step of deciding that the states oftracking of the object and object zone are the stand-alone state if theclassified class includes a single object and also neither of the statesof tracking of the object and the object zone included in the class isthe state of parting.
 47. An object-tracking method according to claim43, wherein said state of tracking includes the state of parting and thecrossover state in which a plurality of objects correspond to a singleobject zone, and said object-tracking method includes a step of decidingthat the states of tracking of the objects and object zone included inthe classified class are the crossover state if the class includes twoor more objects and neither of the states of tracking of the objects andthe object zone included in the class is the state of parting.
 48. Anobject-tracking method according to claim 44, wherein said state oftracking includes the state of parting and the crossover state in whicha plurality of objects correspond to a single object zone, and saidobject-tracking method includes a step of deciding that the states oftracking of the objects and object zone included in the classified classare the crossover state if the class includes two or more objects andneither of the states of tracking of the objects and the object zoneincluded in the class is the state of parting.
 49. An object-trackingmethod according to claim 37, including steps of extracting zonecharacteristic quantities from said image information, said object-zoneinformation and said first zone-correspondence information and providingthe zone characteristic-quantity information, which is the informationindicative of said zone characteristic quantities, storing said objectcharacteristic quantities and selecting the stored object characteristicquantities to be provided as required, and updating said stored objectcharacteristic quantities on the basis of said zonecharacteristic-quantity information, said first zone-correspondenceinformation or the correspondence information that has been determinedand the object characteristic quantities generated prior to the present.50. An object-tracking method according to claim 49, wherein said stateof tracking includes the state of parting that is a transient statethrough which an object zone parts into a plurality of object zones, andsaid object-tracking method includes steps of including, into the zonecharacteristic-quantity information, the information which indicatesthat there is no need to establish corresponding relationship to anyobjects, for the object zones that are indicated as having states otherthan the state of parting while in their tracking states, and excludingthe object zones, in which the zone-correspondence information indicatesthat there is no need of establishing the corresponding relationship toany objects, from the establishment of corresponding relationship. 51.An object-tracking method according to claim 49, wherein said state oftracking includes a stand-alone state in which a single object ispresent in an object zone, and said object-tracking method includessteps of deciding whether or not the state of tracking is thestand-alone state based on said first zone-correspondence information orcorrespondence information that has been determined, and skipping anupdate of the stored object characteristic quantity if the state oftracking of an object is any of the states other than the stand-alonestate.
 52. An object-tracking method according to claim 37, includingsteps of determining all possible combinations of objects and objectzones on the basis of said object characteristic quantities and saidfirst zone-correspondence information, and synthesizing objectcharacteristic quantities to generate synthesized characteristicquantities only for the determined combinations of objects and objectzones.
 53. An object-tracking method according to claim 37, includingsteps of finding synthesis ratios, which are the coefficients foradjusting the ratios of said object characteristic quantities to besynthesized, and generating synthesized characteristic quantities on thebasis of said synthesis ratios and object characteristic quantities. 54.An object-tracking method according to claim 37, including steps ofreceiving zone characteristic quantities together with objectcharacteristic quantities, calculating synthesized characteristicquantities for arbitrary synthesis ratios based on the received zonecharacteristic-quantity information and the object characteristicquantities, and providing the synthesized characteristic quantitycorresponding to the highest degree of similarity between the zonecharacteristic quantity and the calculated synthesized characteristicquantity.
 55. An object-tracking method according to claim 37, whereinsaid state of tracking includes a state of parting, said state ofparting being a transient state through which an object zone parts intoa plurality of object zones, and said object-tracking method includes astep of generating synthesized characteristic quantities only for theobject zones indicated as being in the state of parting while in theirtracking states.
 56. An object-tracking method according to claim 37,wherein said object characteristic quantity includes an area of anobject, and said object-tracking method includes steps of calculating asynthesis ratio, which is a coefficient for adjusting the ratios of theobject characteristic quantities to be synthesized, on the basis of theareas of objects and generating a synthesized characteristic quantityfrom the calculated synthesis ratio and object characteristicquantities.
 57. An object-tracking method according to claim 56, whereinsaid synthesis ratio is restricted within a range predetermined on thebasis of the variations of the object areas.
 58. An object-trackingmethod according to claim 37, including steps of receiving said zonecharacteristic quantities together with said object characteristicquantities, generating synthesized characteristic quantities within therange of variations in the object areas based of the received zonecharacteristic quantities and the object characteristic quantities, andproviding the synthesized characteristic quantity which has the highestdegree of similarity to the zone characteristic quantity of the objectzone of interest.
 59. An object-tracking method according to claim 37,wherein said object characteristic quantity includes an image template,which describes a shape and/or color of an object, and saidobject-tracking method includes steps of deciding the back-to-bellyrelations of the objects on the basis of the image templates and zonecharacteristic quantities, and synthesizing the image templates based onsaid decided back-to-belly relations of the objects to obtain asynthesized characteristic quantity.
 60. An object-tracking methodaccording to claim 37, including steps of calculating, based on saidsynthesized characteristic-quantity information, said zonecharacteristic-quantity information and said first zone-correspondenceinformation, the combination of objects and object zones which has thehighest degree of similarity from all the possible combinations ofobjects and object zones that can be associated in correspondingrelationship, selecting the calculated combination of objects and objectzones as the objects and object zone of optimum correspondence, andgenerating the optimum-correspondence information that indicates theoptimum correspondence relation of the objects and object zone, anddetermining corresponding relationship between objects and object zonesbased on said first zone-correspondence information and saidoptimum-correspondence information and providing the correspondenceinformation that has been determined, which is the information inclusiveof the determined corresponding relationship between objects and objectzones.
 61. An object-tracking method according to claim 60, includingsteps of calculating a total degree of similarity for all possiblecombinations of objects and object zones, wherein said total degree ofsimilarity is a sum of the degrees of similarity between characteristicquantities of object zones and synthesized characteristic quantities ineach of the combinations, and deciding that the combination of thehighest total degree of similarity of all said possible combinations isthe combination of the highest similarity.
 62. An object-tracking methodaccording to claim 60, wherein said first zone-correspondenceinformation includes the information about an at-rest/in-motion statethat indicates whether an object zone is at rest or in motion, and saidobject-tracking method includes a step of excluding, from all thepossible combinations of objects and object zones, the combination ofthe object and object zone indicated as being in the at-rest state bysaid information about an at-rest/in-motion state.
 63. Anobject-tracking method according to claim 60, including steps of: if thedegrees of combined similarity, obtained from the degrees of similarityof the sets of objects and an object zones that make up the combinationdecided as a combination of the highest similarity, is equal to or lowerthan a predetermined threshold, then selecting the combinations of thedegrees of combined similarity within a predetermined threshold from thecombinations of the degrees of combined similarity ranked as the highestsimilarity, of all the possible combinations of objects and objectzones, including the corresponding relationship between objects andobject zones common to the selected combinations into theoptimum-correspondence information as optimum correspondences, andfurther including the information that indicates absence of optimumcorrespondence between any object and object zone in theoptimum-correspondence information, for the object and object zone inthe correspondence relation that is not included in the correspondingrelationship of the objects and object zones common to said selectedcombinations; for the objects not indicated in saidoptimum-correspondence information that are absent from the optimumcorresponding relationship to object zones, providing the informationindicating the corresponding relationship between the objects ofinterest and object zones included in said optimum-correspondenceinformation, as the correspondence information that has been determined;and for the objects indicated in said optimum-correspondence informationthat are absent from the optimum corresponding relationship to objectzones, providing the information indicating the correspondingrelationship between the objects of interest and object zones includedin the first zone-correspondence information, as the correspondenceinformation that has been determined.
 64. An object-tracking methodaccording to claim 61, including steps of: if the degrees of combinedsimilarity, obtained from the degrees of similarity of the sets ofobjects and object zones that make up the combination that is decided asthe combination of the highest similarity, is equal to or lower than apredetermined threshold, then selecting the combinations of the degreesof combined similarity within a predetermined threshold from thecombinations of the degrees of combined similarity ranked as having thehighest similarity, from all the possible combinations of objects andobject zones, including the corresponding relationship between objectsand object zones common to the selected combinations in theoptimum-correspondence information as optimum correspondences, andfurther including the information that indicates absence of an optimumcorrespondence between any object and object zone in theoptimum-correspondence information, for the object and object zone inthe correspondence relation that is not included in the correspondingrelationship of the objects and object zones common to said selectedcombinations; for the objects not indicated in saidoptimum-correspondence information that are absent from the optimumcorresponding relationship to object zones, providing informationindicating the corresponding relationship between the objects ofinterest and object zones included in said optimum-correspondenceinformation, as the correspondence information that has been determined;and for the objects indicated in said optimum-correspondence informationthat are absent from the optimum corresponding relationship to objectzones, providing the information indicating the correspondingrelationship between the objects of interest and object zones includedin the first zone-correspondence information, as the correspondenceinformation that has been determined.
 65. An object-tracking methodaccording to claim 60, wherein said state of tracking includes a stateof parting, said state of parting being a transient state through whichan object zone parts into a plurality of object zones, and saidobject-tracking method includes a step of determining the correspondingrelationship between objects and object zones to be identical with thoseindicated in the optimum-correspondence information only for the objectzones that are indicated as having the state of parting while in theirtracking states.
 66. An object-tracking method according to claim 60,wherein said state of tracking includes a state of parting, said stateof parting being a transient state through which an object zone partsinto a plurality of object zones, and said object-tracking methodincludes a step of providing the correspondences between objects andobject zones included in the first zone-correspondence information asthe correspondence information that has been determined only for theobject zones indicated as having the state other than the state ofparting while in their tracking states.
 67. An object-tracking programfor tracking an object based on image information, said programoperating a computer to execute processes of receiving imageinformation, synthesizing object characteristic quantities thatrepresent characteristic quantities of respective objects included insaid received image information and generating a synthesizedcharacteristic quantity, and establishing a correspondence between saidobjects and an object zone based on the degree of similarity between acharacteristic quantity of said object zone and said synthesizedcharacteristic quantity, said object zone being a region that isextracted from said image information and also includes said objects.68. An object-tracking program for establishing correspondences betweenobjects and object zones included in received image information, saidprogram operating a computer to execute processes of receiving imageinformation, extracting object zones from said received imageinformation and providing the object-zone information inclusive of imageinformation about said object zones, deciding the state of tracking witheach object or object zone on the basis of said object-zone informationand the correspondence information that has been determined indicatingcorresponding relationship of the objects and object zones prior to thepresent and providing first zone-correspondence information, whichindicates the corresponding relationship of said object zones andobjects and the states of tracking, generating the zone characteristicquantities, which represent the characteristic quantities of the objectzones, and the object characteristic quantities, which represent thecharacteristic quantities of the objects through the use of said imageinformation, said object-zone information and said firstzone-correspondence information, synthesizing characteristic quantitiesfor all required combinations of a plurality of objects to generate eachof synthesized characteristic quantities based on said objectcharacteristic quantities and said first zone-correspondenceinformation, and providing synthesized characteristic-quantityinformation, which is the information that includes said synthesizedcharacteristic quantities and the corresponding relationship between theobjects used for generating synthesized characteristic quantities andthe synthesized characteristic quantities, and associating said objectsand object zones in corresponding relationship based on said firstzone-correspondence information, said zone characteristic-quantityinformation and said synthesized characteristic-quantity information,and providing said correspondence information that has been determinedfor the present.
 69. An object-tracking program for establishingcorrespondences between objects and object zones included in receivedimage information, said program operating a computer to executeprocesses of receiving image information, deciding the state of trackingeach object or each object zone on the basis of said object-zoneinformation and the correspondence information that has been determinedindicating the corresponding relationship of the objects and objectzones prior to the present, and providing first zone-correspondenceinformation, which indicates the corresponding relationship of saidobject zones and objects and the states of tracking, generating the zonecharacteristic quantities, which represent the characteristic quantitiesof the object zones, and the object characteristic quantities, whichrepresent the characteristic quantities of the objects through the useof said image information, said object-zone information and said firstzone-correspondence information, and while taking each of the objects asa target, deciding the correspondences between said objects and objectzones through designating the correspondences between objects and objectzones included in said first zone-correspondence information as saidcorrespondences between said objects and object zones, for the objectsthat are decided to be in states other than a state of parting while intheir tracking states, said state of parting being a transient statethrough which an object zone parts into a plurality of object zones; forthe objects that are decided to be in the state of parting while intheir tracking states, synthesizing characteristic quantities for allrequired combinations of a plurality of objects on the basis of saidobject characteristic quantities and said first zone-correspondenceinformation to generate respective synthesized characteristicquantities; and comparing each of synthesized characteristic quantitieswith a zone characteristic quantity to associate the objectscorresponding to the synthesized characteristic quantity, which has thehighest degree of similarity to an object zone, with the object zone ofinterest to be placed in corresponding relationship.